The SiRFnavIV API protocol

[E3V3A]

Hello!
I found your interesting project on Replicant. I'd love to see a proper GPS decoder for the GT-I9100 and others. Have you been able to decode this yet? Also, did you see THIS document? Also I found another GithUb repo with some info on the SiRF binary mode (aka OSP).

The A2235-H is based on CSR SiRFstarIV chipset, which supports a novel GPS fix mode and a variety of low power states.
The A2235-H supports two different data modes: NMEA and SiRF binary mode (also called OSP mode). Technical Machine's default APIs use the module in NMEA mode.

From THIS manual:

10.3 SiRFawareTM Support
SiRFawareTM is a low-power operating mode that seeks to maintain low uncertainty in position, time, and frequency, and to maintain valid current Ephemeris using either data collected from satellites in view or Extended Ephemeris methods. The SiRFawareTM mode is entered using the One Socket Protocol, an extension of the SiRF Binary Protocol. Please refer to the appropriate manual. In order to request a fix and to exit SiRFawareTM it is necessary to toggle the ON_OFF pin. Toggling is done by pulling the signal to HIGH for about 200ms.

[mnalis]

Thanks for your interest! Unfortunately, GPS in GT-I9100 does not output in SiRF Binary protocol (OSP or older SSB) - or at least nobody ever found out how to make them do it. Neither does it output in NMEA 0183. So the docs on OSP/NMEA don't help unfortunately. If it generated OSP (or NMEA), open source http://www.catb.org/gpsd/ would be able to parse them and provide GPS functionality (and it doesn't).

So idea of this project is to try to find out what exactly is GSD4t in GT-9100 outputting (it is some other binary protocol, not OSP), and then make a converter to NMEA (or OSP). If that works, adding the support to gpsd(8) will be relatively easy.

I've making some progress in decoding, but still nothing definitive. The front-page README.md will be updated when some significant breakthrough occurs (as will Replicant forum thread).

[E3V3A]

@mnalis Ah, thank you for clarifying. Because from the website for GSD4t, I was under the impression that one could switch the output format by sending certain commands. If you have more HW details, I may be able to help you search. Perhaps we can dump the ROM and reverse from there? BTW. How can I find the correct device on the SlimKat 4.4.2b4 to connect with, and how?

[mnalis]

Yes, some of the chipsets seem to be capable to switch from OSP to NMEA mode and back. Unfortunately the GSD4t on GT-9100 does not run in either mode so OSP or NMEA commands could not be used; and if there exist such command in it's own binary protocol (for which is there no documentation) it is unknown.

I suspect it depends on the firmware running on the GPS chip. Versions of HW and all details I have can be found in debug logs https://github.com/mnalis/gsd4t_logs/blob/master/strace6-rst-short/SLCLog.gp2.parsed.txt

So while it might be possible to find other GPS firmware that would produce NMEA or OSP on that GSD4t, it might also brick it - so I've decided on non-intrusive read-only decoding effort.

As for SlimKat, I haven't used it, but to find what device is GPS (so you can strace it as described in https://github.com/mnalis/gsd4t_parser/blob/master/README.txt about line 178) you should do something like

adb shell
su
cd /data
lsof | grep /dev/tty > gpsoff.txt

start your GPS application at this time before proceeding with commands!

lsof | grep /dev/tty > gpson.txt
diff gpsoff.txt gpson.txt

if SlimKat is anything like Cyanogenmod, the GPS device will only be opened when used, so diff above would show you which new devices were opened indicated with "+" or ">" sign (with any luck, there would be only one, like /dev/ttySAC1 in my case.

[E3V3A]

@mnalis Thanks, that was a good tips, also for finding other devices.

[E3V3A]

Yes, I'm running on SlimKat KK4.4.2b4 and found it under the same device as you. It's funny, it is very sensitive and stops output shortly, after dumping. Then I have to restart the GPS app. I use GPS Status. I tried dumping with: cat /dev/ttySAC1 |hexdump -C and it always ends with:

...
00004d60  00 43 03 00 00 1a 00 4b  00 1d 00 01 00 0f c6 e6  |.C.....K........|
00004d70  e1 09 5e 09 0d 08 47 50  53 20 53 74 6f 70 00 f1  |..^...GPS Stop..|
00004d80  e4 b0 b3 a0 a2 00 00 1e  1f 00 01 00 11 0e 1f 82  |................|
00004d90  30 17 04 01 00 f9 c5 68  00 47 53 34 74 40 01 01  |0......h.GS4t@..|

[E3V3A]

@mnalis Is there any simple way to parse any of this from within phone? Also, check new comment in Replicant forum... Also, can you say something more about all the log files. It's hard to know what is the raw output (as from above) and what you have manipulated (and how)?

[mnalis]

cat /dev/ttySAC1 |hexdump -C exits because cat(1) will exit when it finds EOF (no more data at that moment). To parse data continuously, you would need instead tail -f /dev/ttySAC1 |hexdump -C, or more properly hexdump -Cv /dev/ttySAC1. However, this wouldn't work correctly (as you've seen) -- because when hexdump reads from device, that data won't reach your GPS application - and your GPS application will timeout and try restarting GPS suspecting lockup (which is what you see with 'GPS stop' text in your hexdump). So, no - there isn't simple way to parse it from within phone.

Only non-invasive way is using strace on the phone (as detailed in README.txt in "strace_to_gp2.pl" section and below), then transfer that strace.log to the computer with perl (I'm running Debian GNU/Linux here), and then do ./strace_to_gp2.pl strace.log | ./enc4t_gp2_to_raw4t_gp2.pl | ./raw4t_gp2_to_human.pl which will parse as much as possible in human readable form (and give you rest as hex codes). And then compare that output with ./sirfbin_gp2_to_human.pl SLCLog.gp2 to see what more data you can match to decode the GSD4t protocol.

Or, if you lack hardware/software/time, you can simply take example log files from github and compare (for example, https://github.com/mnalis/gsd4t_logs/blob/master/strace6-rst-short/SLCLog.gp2.human.txt with https://github.com/mnalis/gsd4t_logs/blob/master/strace6-rst-short/strace.log.human.txt)

First one (SLCLog.gp2.human.txt) is in parsed SiRF OSP format, which is understood by gpsd and output by closed-source android software. Second one (strace.log.human.txt) is raw output from GPS chip, with annotated parts where I've been able to match or guess the meaning of data. So, when there is no more unknown bytes (or, at least, no more IMPORTANT unknown bytes) in strace.log.human.txt the new binary protocol is understood, and one can start writing new parser for gpsd which could then work with GSD4t without closed-source software.

[E3V3A]

Great work!

exits because cat(1) will exit when it finds EOF (no more data at that moment).

Ahh, of course! Thanks for reminding me. Easy to forget, and also explains other funny behavior I've seen on other devices recently. So I have been looking at your logs and it seem that yo have solved pretty much everything. Is that correct?

I have also looked after some more documentation. And found THIS very useful manual. Then there is the code of the driver_sirf.c that could be useful to look at, because of its very nice comments. Especially line ~367. Which seem to suggest that the unknown sequences are some kind of mode change, but I'm not sure. For example, compare to page 106, where is suggests that last 2 bytes of payload, before the check-sum and End-Seq, is a bit rate.

Oh, and another thing. Have you tried to sniff the network during the operation. It's possible that some of those sequences are binary ephemeris or almanac data, that is either downloaded from SGEE servers or from cache.

<01:13:57> *** [SGEE SERVER1 : sgee.samsung.csr.com]
<01:13:57> *** [SGEE SERVER2 : instantfix.csr.com]
<01:13:57> *** [SGEE PORT : 80]
<01:13:57> *** [SGEE FILE  :/diff/packedDifference.f2p3enc.ee]
<01:13:57> *** [SGEE URID : 1]
<01:13:57> *** [SGEE DEVICE ID : 00 00 00 00 00 00 00 02 ]
<01:13:57> *** [SGEE OEM ID : 00 00 00 00 ]

[E3V3A]

This document on the Fastrax IT430 may also be useful.

Default protocol for host communication is NMEA 4800 baud, ref (2). Protocol is switchable to SiRF binary OSP (One Socket Protocol, ref (3) by NMEA protocol command $PSRF100.

Default NMEA message configuration: $GPGGA, $GPGSA and $GPRMC rate every second (in this order) and $GPGSV messages (can be 1… 4) every 5 seconds (sent after $GPGSA message). Also CGEE related $PSRF156 messages can be outputted at irregular interval. Message output and rate can be configured by NMEA message $PSRF103. $PSRF150 (OKtoSend) messages are also sent out when the receiver is ready to receive messages after wake up or when it is going into low-power mode and can’t process input commands anymore.

[E3V3A]

I just found a much more recent version of the OSP protocol, where some things seem to have changed, since the doc I posted above.

[mnalis]

Thanks for encouragement... No, while I have matched many of the messages, the few big ones (and unfortunately complicated ones) still remain, and I have a feeling they have the most important data. Sniffing the network is a good idea, but I'm currently only looking at downloading data from chip (not AGPS data which will be uploaded to the chip after being pulled from the Internet or cache).

as for the OSD commands and NMEA, those unfortunately doesn't help as GSD4t don't speak either.

[E3V3A]

as for the OSD commands and NMEA, those unfortunately doesn't help as GSD4t don't speak either.

(You mean OSP?) Then I assume you didn't actually bother to look at those documents. I have read 3 different documents/manuals, that all state that the GSD4t talk both OSP and NMEA when set properly. The Fastrax above, is based on that same chip. (I can help finding and reading documents, but I can't help when people are being stubborn.) We could both still be right, since "our" chip could have a different FW, from that found in other OTC sold chips.

So what is your plan then? As I see it, there are only 3 options:

1. Sniff the communication between AOS and GSD4t
2. RE the GSD4t FW (we need to extract it first, unless we can find an update blob.)
3. RE the GSD4t libraries used in AOS.

Also, what is it that make you believe that those remaining messages are more important? What are you missing?

[mnalis]

yes, I mean OSP, that was typo, sorry.

Yes, the GSD4t solution does speak both OSP and NMEA, that is true. But the GSD4t chip does not. I'll try to explain.

In past, we had SiRFstar III and like, which contained on themselves firmware which would speak NMEA and/or SiRF OSP via serial interface to the OS. Today, GSD4e works that way too.

However, GSD4t was entry level (cheaper) solution, which means it does part of the job on host CPU (that is, same ARM Cortex-A9 CPU that the Android OS itself and all Android apps run on). Here is what the makers themselves say:

A GSD4t solution comprises a GSD4t hardware implementation with SiRFHost™ software running on a host CPU.

Which means you send SiRF OSP (or NMEA) commands to the "SiRFHost" software package (and others: SiRFInstantFix, SiRFLSM) runing on your main CPU, and not to the chip itself (as you would with GSD4e).

So, as you can see, it is unfortunate that SiRF/CSR decided to call under the same name "gsd4t" two completely different things (both hardware chip that talks one unknown binary protocol only, and hardware+software "solution" which speaks standard NMEA and/or OSP)

However, the goals of Replicant are to remove all non-free closed source software from host CPU. That means removing "SiRFHost" libraries (contained in /system/lib/hw/gps.exynos4.so), and, hence, removing OSP and NMEA support. So, what I set to do, is ultimately write free software replacement for "SiRFHost" libraries running on main CPU. And for that, I have to understand the unknown internal binary protocol which is being spoken between SiRFHost libraries and raw GPS chip (via /dev/ttySAC1 serial interface).

Note that while it would be indeed very nice if we could remove all traces of non-free firmwares and thus liberate non-host CPUs, that is way way harder (as you don't even know what hardware it that firmware running on, you don't even have dissasemblers for it etc) so it borders on impossible.

I hope I've managed to explain how GSD4t in Samsung Galaxy S2 speaks OSP and NMEA, and how we cannot use that for free software replacement (except as a debugging help during reverse engineering the unknown binary protocol).

[mnalis]

So what is your plan then? As I see it, there are only 3 options:

1. Sniff the communication between AOS and GSD4t
2. RE the GSD4t FW (we need to extract it first, unless we can find an update blob.)
3. RE the GSD4t libraries used in AOS.

primarily, I intent to continue doing (1) (from strace ouput) and document and extract as much as possible information in human readable form. (currently the output is pretty coarse and generic).

After that, I (or hopefully someone else would jump in) should learn a lot more up on general GPS operations (also http://www.catb.org/gpsd/gps-hacking.html, http://www.catb.org/gpsd/references.html) and then try to see how I can calculate lat/lon from "pseudorange measurements" which I've hopefully managed to document in (1). If that cannot be acomplished (either because math is too complex for me, or because I didn't manage to extract enough useful data in step (1)), then it will be necessary to try (3) -- reverse engineer /system/lib/hw/gps.exynos4.so and try to see how it does it. Note that I'd like to avoid that reverse engineering effort, because that chunk of aseembly is HUGE.

(2) is probably hopeless (see comments about liberating firmwares in comment above).

Also, what is it that make you believe that those remaining messages are more important? What are you missing?

well, they are the biggest blobs and start happening about the time I get GPS fix, so I assume they might contain "pseudorange measurements" (whatever are those EXACTLY - I need to learn on GPS implementetion details, see above). But, I don't know enough about GPS tech - it might be I already have the data needed to trilaterate pseudoranges to coordinates, but I don't know that.

As for help - the biggest one would be someone knowing enough GPS math and teminology to say what data is needed to trilaterate, and what of the data in the hex dumps looks like it might be it.
I've found some code snippets which might help in calculations and verification (like https://www.ngs.noaa.gov/gps-toolbox/bc_velo.htm and http://home-2.worldonline.nl/samsvl/stdalone.htm and http://home-2.worldonline.nl/~samsvl/software.htm -- more such software might be helpful, especially if written in perl or C, but I'll parse most things probably. Someone willing to try that and see what results they got would be most helpful too).

other than that, help matching the raw data to the SiRFHost debug messages and writing more detailed human readable descriptions would be nice. Some perl knowledge needed.

Most of all, I'd love more free time -- but that nobody can help :)

[E3V3A]

Hi, thanks for that explanation. I have since understood quite a bit more about this. Right, now. I'll just keep it short. I was looking again at your unknown MID 0x45 (69) which seem undocumented at first, but then there is a small note on MID 232 (0xE8) "Extended Ephemeris" is saying that its SID (0xD0) is equivalent to MID:SID of 0x45:0x3...

MID:0x45   MID_POS_MEAS_RESP 
SID:0x01   POS_RESP X
SID:0x02   MEAS_RESP

Can you also add the SID byte to your parser? (That should be the byte directly after the MID.) And I guess that not all OSP has neither MID nor SID. (See OSP-ICD manual.)

The payload always starts with a 1-byte Message ID (MID) field. Depending on the MID value, a 1-byte Sub ID (SID) field may follow it. Subsequently, and again depending on the value of the MID field and the value of the SID field if it exists, a variable number of message parameter fields follow.

Second, I misread older document about maximum payload length. It should be 2^11-1=2047 (0x7FF) bytes.

[mnalis]

Can you also add the SID byte to your parser? (That should be the byte directly after the MID.) And I guess that not all OSP has neither MID nor SID. (See OSP-ICD manual.)

True, not all MIDs have SIDs. SID (if present) is already shown as first byte of hex dump payload, for example in skip unknown MID 0x45 -- hex 02 00 01 04 00 00 00 00 00 00 00 00 the SID (if it exists for such MID) would be 0x02

I've added MID 0x45 SID1-2 (guessed them from DetailedLog.txt too)

Note, I've also found another SiRF parser so I've added it at https://github.com/mnalis/gps-kit-cvs and fixed a few minor things. It doesn't support many MIDs, but have detailed output about those it does support - see https://github.com/mnalis/gsd4t_logs/blob/master/strace6-rst-short/SLCLog.gp2.human2.txt

[E3V3A]

Any news or updates?

[illarionov]

Yet another parser (my), for completeness: https://github.com/illarionov/sirfdump

[fcancillo]

Good morning.
I just put a repository named RXtoRINEX where there are some tools to collect / convert OSP data to RINEX.
Although RINEX is not aimed to real time processing, but to post-processing for high accuracy positioning, the important point is that RINEX files contains all data need for computing position solutions. To generate a RINEX observations file from OSP data they are needed MID28 and MID7 to generate epoch data, and MID2, MID6 and MID7 would be useful to get some header data. To generate a RINEX navigation file they are need: a MID15 (complete ephemerides data collected by the receiver) or the MID8 (50 bps GPS message data) for all satellites being tracked.
The conclusions are:
• If you only need the position solution computed by the receiver, MID2 provides you X,Y,Z coordinates. Alternatively you have MID41 with latitude, longitude and altitude.
• If you plan to compute your own solution from GPS data, you should need MID28 (observables for a satellite) for all satellites being tracked, MID7 with the clock corrections, and satellite orbital ephemerides, which can be obtained from MID8 or MID15.
• If you plan to compute an improved solution, you should need additional data from other sources, like GPS augmentation data provided by a SBAS (WAAS, EGNOS or MSAS), or like NTRIP data being provided by some reference stations networks.
With regard to data I collected from SiRFIV receivers, from a GSD4e based G-STARIV it was possible to collect all data to generate RINEX files, but from a GSD4t based Samsung Galaxy S2 only observation data were included in the GP2 file.
May be the above mentioned project contents could by useful for your objectives.

[moshp1]

Hi everyone .
I followed this very informative forum for a while and enjoyed the high level(though too high sometimes for me..). now posting my first message.
I will be happy to get help with a big problem .
im using a moudle which integrating the gsd4e chip . I am trying to builed it my self kind of , but the first problem in the way there is that I could not find any refernce design of the gsd4e chip it self.
is it posible to find it ? did some one try to use the GSD4E on its own ?

thank you very much
moshp1

[fcancillo]

Good evening,
If you plan to use this GPS receiver in a board you are designing, you should ask CSR (http://www.csr.com/products/35/sirfstariv-gsd4e) or a near distributor for chips available and specifications (may be they do not pay attention if you request few chips). If you are developing a prototype or you would manufacture few boards, reconsider the possibility to use a module (GlobalSat, Fastrax, etc.) because it would integrate in a chip size device the SiRF receiver, clock, RF I/F, etc., they should be easier to obtain, and specs are available.
At system/software level, the interfaces with the receiver are described in the SiRFstartIV OSP ICD (if you plan to used the binary I/F), and in the NMEA Reference Manual (if you plan to use this I/F).

[moshp1]

Hi fcancillo!

thank you for your answer .

yes i'm aware of modules avilable , and even tried one to see it feets me. it does. the problem is that I have some difficullties to integrate the moudle for my own reasons, thus I need the gsd4e/gsd4t itself.
the reason I asked here for help is exactly because what you mentioned - they (CSR = Qualcom) do not pay attention to little customers like me , and it is not possible to desing with out the full data sheet (in there web only a portfolio is available) .
so if some does have it by chance , or any schematic ref design , I would really appreciate it.
thank you !

[zamaudio]

This might be useful cough https://github.com/x893/SiRFLive

[mirh]

https://forum.xda-developers.com/t/gps-sirfiv-tool-development-disable-static-navigation.1324219/page-5
https://forum.xda-developers.com/t/mod-settings-tweak-sirf-gps-on-sgs2.1322765/
https://github.com/zp8001/STUDY_4.0.3/blob/master/android/hardware/nufront/gst4tq_gb/app/AndroidSharedLib/source/gps_sirf.c