Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dump1090 improvements. Overhauled traffic parsing. UAT groundspeed and air/ground status bug fixes. #285

Merged
merged 12 commits into from Feb 26, 2016

Conversation

ghost
Copy link

@ghost ghost commented Feb 24, 2016

Add Stratux-specific branch of MalcolmRobb/dump1090 and overhaul the traffic.go parsing routines.

Dump1090 changes:

  • Add parsing of 1090ES TIS-B messages
  • Add parsing of ADS-B / ADS-R / TIS-B NACp category from DF=17, TC=29/31 and DF=18, TC=19 messages
  • Add parsing of GNSS altitude flag
  • Add parsing of delta height between GNSS altitude and baro altitude
  • New JSON-formatted output of fields used by traffic.go are sent on TCP port 30006. These include additions for NACp, signal strength, delta height, and GNSS height flag as noted in the dump1090 README.

Stratux changes:
1090 ES parsing is completely overhauled to support the new dump1090 format. Additional ES parsing changes include:

  • dump1090 raw signal strength is reported
  • NIC is now calculated from DF17 position message type code
  • NACp is reported when available (else, it is set to NIC, because not all emitters send this characteristic)
  • Addr_type (UAT address qualifier) is set per the presence of DF=17 or DF=18 with the appropriate Control Field flag. This will allow GDL90-connected devices to identify whether targets are ADS-B, ADS-R, or TIS-B
    -New TargetType flag will allow traffic source to be cleanly shown on UI traffic page (future commit)

978 UAT parsing improved to fix a couple of significant bugs:

  • ti.OnGround initializes to 'false' when new ti is created, but is never explicitly set to false when vaild airborne messages are received. If a 978 target is ever seen on the ground, it will never be set to flight status.
  • ew_vel and ns_vel changed from int16 to int32 to solve the zero airspeed issue noted in UAT rebroadcast traffic speed incorrect #277

Other 978 improvements include:

  • NIC and NACp set from UAT message
  • UAT signal strength is now passed to the traffic decoder and is tracked as part of each traffic target (similar to changes for 1090ES)

Refinements to GDL90 message packaging include:

  • Track type set to "true track" rather than "true heading" (latter applies mainly to surface vehicles).
  • Add support for setting "Extrapolated position" flag for future traffic coasting algorithm
  • Report actual NIC / NACp of targets, rather than hardcoded "1"

Other changes include:

  • Traffic database is printed to stratux.log every ten seconds as a debug / troubleshooting aid. This frequency may be reduced or turned into a selectable debug option in future releases.
  • Add tracking of delta height between GNSS altitude and baro altitude. This will eventually be used to feed this to a regression function that will estimate pressure altitude from GPS height.
  • Added fields to support future target coasting.
  • Added fields to support future target relative distance and bearing calculation

To-do items include:

  • Traffic UI updates to show TIS-B / ADS-R status
  • Target coasting
  • Target relative distance and bearing
  • Apply dump1090 improvements to dump1090-mutability, which is an actively-developed branch with numerous improvements that may improve 1090ES performance.
2016/02/24 04:05:30 List of all aircraft being tracked:
2016/02/24 04:05:30 ==================================================================
2016/02/24 04:05:30 AB4EEB => {"Icao_addr":11226859,"Tail":"","Emitter_category":0,"OnGround":false,"Addr_type":0,"TargetType":0,"SignalLevel":4,"Position_valid":false,"Lat":0,"Lng":0,"Alt":0,"GnssDiffFromBaroAlt":0,"AltIsGNSS":false,"NIC":0,"NACp":0,"Track":0,"Speed":0,"Speed_valid":false,"Vvel":0,"Timestamp":"2016-02-24T04:05:26.369Z","Age":3.3,"Last_seen":"0001-01-01T00:00:15.8Z","Last_alt":"0001-01-01T00:00:00Z","Last_GnssDiff":"0001-01-01T00:00:00Z","Last_GnssDiffAlt":0,"Last_speed":"0001-01-01T00:00:00Z","Last_source":1,"ExtrapolatedPosition":false,"Bearing":0,"Distance":0}
2016/02/24 04:05:30 A8D273 => {"Icao_addr":11063923,"Tail":"FDX1231","Emitter_category":0,"OnGround":false,"Addr_type":0,"TargetType":1,"SignalLevel":17,"Position_valid":true,"Lat":44.567,"Lng":-95.30189,"Alt":37000,"GnssDiffFromBaroAlt":-500,"AltIsGNSS":false,"NIC":8,"NACp":8,"Track":158,"Speed":469,"Speed_valid":true,"Vvel":0,"Timestamp":"2016-02-24T04:05:30.596Z","Age":1,"Last_seen":"0001-01-01T00:00:20.05Z","Last_alt":"0001-01-01T00:00:20.05Z","Last_GnssDiff":"0001-01-01T00:00:17.35Z","Last_GnssDiffAlt":37000,"Last_speed":"0001-01-01T00:00:17.35Z","Last_source":1,"ExtrapolatedPosition":false,"Bearing":0,"Distance":0}
2016/02/24 04:05:30 A7E5FD => {"Icao_addr":11003389,"Tail":"","Emitter_category":0,"OnGround":false,"Addr_type":0,"TargetType":0,"SignalLevel":12,"Position_valid":false,"Lat":0,"Lng":0,"Alt":37000,"GnssDiffFromBaroAlt":0,"AltIsGNSS":false,"NIC":0,"NACp":0,"Track":0,"Speed":0,"Speed_valid":false,"Vvel":0,"Timestamp":"2016-02-24T04:05:30.323Z","Age":15.4,"Last_seen":"0001-01-01T00:00:03.7Z","Last_alt":"0001-01-01T00:00:19.75Z","Last_GnssDiff":"0001-01-01T00:00:00Z","Last_GnssDiffAlt":0,"Last_speed":"0001-01-01T00:00:00Z","Last_source":1,"ExtrapolatedPosition":false,"Bearing":0,"Distance":0}
2016/02/24 04:05:30 A0678C => {"Icao_addr":10512268,"Tail":"1200","Emitter_category":1,"OnGround":true,"Addr_type":0,"TargetType":1,"SignalLevel":293,"Position_valid":true,"Lat":44.829132,"Lng":-93.456116,"Alt":975,"GnssDiffFromBaroAlt":0,"AltIsGNSS":false,"NIC":9,"NACp":10,"Track":99,"Speed":16,"Speed_valid":true,"Vvel":0,"Timestamp":"2016-02-24T04:05:28.705115282Z","Age":0.9,"Last_seen":"0001-01-01T00:00:18.2Z","Last_alt":"0001-01-01T00:00:18.2Z","Last_GnssDiff":"0001-01-01T00:00:00Z","Last_GnssDiffAlt":0,"Last_speed":"0001-01-01T00:00:18.2Z","Last_source":2,"ExtrapolatedPosition":false,"Bearing":0,"Distance":0}
2016/02/24 04:05:30 A40D8D => {"Icao_addr":10751373,"Tail":"","Emitter_category":0,"OnGround":false,"Addr_type":0,"TargetType":0,"SignalLevel":9,"Position_valid":false,"Lat":0,"Lng":0,"Alt":4050,"GnssDiffFromBaroAlt":0,"AltIsGNSS":false,"NIC":0,"NACp":0,"Track":0,"Speed":0,"Speed_valid":false,"Vvel":0,"Timestamp":"2016-02-24T04:05:30.146Z","Age":13.75,"Last_seen":"0001-01-01T00:00:05.35Z","Last_alt":"0001-01-01T00:00:19.6Z","Last_GnssDiff":"0001-01-01T00:00:00Z","Last_GnssDiffAlt":0,"Last_speed":"0001-01-01T00:00:00Z","Last_source":1,"ExtrapolatedPosition":false,"Bearing":0,"Distance":0}
2016/02/24 04:05:30 A48A83 => {"Icao_addr":10783363,"Tail":"","Emitter_category":0,"OnGround":false,"Addr_type":0,"TargetType":0,"SignalLevel":12,"Position_valid":false,"Lat":0,"Lng":0,"Alt":11425,"GnssDiffFromBaroAlt":0,"AltIsGNSS":false,"NIC":0,"NACp":0,"Track":0,"Speed":0,"Speed_valid":false,"Vvel":0,"Timestamp":"2016-02-24T04:05:30.647Z","Age":10.95,"Last_seen":"0001-01-01T00:00:08.15Z","Last_alt":"0001-01-01T00:00:19.95Z","Last_GnssDiff":"0001-01-01T00:00:00Z","Last_GnssDiffAlt":0,"Last_speed":"0001-01-01T00:00:00Z","Last_source":1,"ExtrapolatedPosition":false,"Bearing":0,"Distance":0}
2016/02/24 04:05:30 AB9A32 => {"Icao_addr":11246130,"Tail":"DAL1605","Emitter_category":0,"OnGround":false,"Addr_type":0,"TargetType":1,"SignalLevel":16,"Position_valid":true,"Lat":44.98366,"Lng":-93.81768,"Alt":15075,"GnssDiffFromBaroAlt":-500,"AltIsGNSS":false,"NIC":8,"NACp":8,"Track":284,"Speed":358,"Speed_valid":true,"Vvel":2944,"Timestamp":"2016-02-24T04:05:28.545Z","Age":1.1,"Last_seen":"0001-01-01T00:00:18Z","Last_alt":"0001-01-01T00:00:18Z","Last_GnssDiff":"0001-01-01T00:00:13.85Z","Last_GnssDiffAlt":14825,"Last_speed":"0001-01-01T00:00:13.85Z","Last_source":1,"ExtrapolatedPosition":false,"Bearing":0,"Distance":0}
2016/02/24 04:05:30 A35A92 => {"Icao_addr":10705554,"Tail":"","Emitter_category":0,"OnGround":false,"Addr_type":0,"TargetType":0,"SignalLevel":22,"Position_valid":false,"Lat":0,"Lng":0,"Alt":18825,"GnssDiffFromBaroAlt":0,"AltIsGNSS":false,"NIC":0,"NACp":0,"Track":0,"Speed":0,"Speed_valid":false,"Vvel":0,"Timestamp":"2016-02-24T04:05:30.576Z","Age":15.8,"Last_seen":"0001-01-01T00:00:03.3Z","Last_alt":"0001-01-01T00:00:18.55Z","Last_GnssDiff":"0001-01-01T00:00:00Z","Last_GnssDiffAlt":0,"Last_speed":"0001-01-01T00:00:00Z","Last_source":1,"ExtrapolatedPosition":false,"Bearing":0,"Distance":0}
2016/02/24 04:05:30 A54C42 => {"Icao_addr":10832962,"Tail":"","Emitter_category":0,"OnGround":false,"Addr_type":0,"TargetType":0,"SignalLevel":9,"Position_valid":false,"Lat":0,"Lng":0,"Alt":13325,"GnssDiffFromBaroAlt":0,"AltIsGNSS":false,"NIC":0,"NACp":0,"Track":0,"Speed":0,"Speed_valid":false,"Vvel":0,"Timestamp":"2016-02-24T04:05:28.877Z","Age":2.3,"Last_seen":"0001-01-01T00:00:16.8Z","Last_alt":"0001-01-01T00:00:18.3Z","Last_GnssDiff":"0001-01-01T00:00:00Z","Last_GnssDiffAlt":0,"Last_speed":"0001-01-01T00:00:00Z","Last_source":1,"ExtrapolatedPosition":false,"Bearing":0,"Distance":0}
2016/02/24 04:05:30 A448DB => {"Icao_addr":10766555,"Tail":"","Emitter_category":0,"OnGround":false,"Addr_type":0,"TargetType":0,"SignalLevel":7,"Position_valid":false,"Lat":0,"Lng":0,"Alt":0,"GnssDiffFromBaroAlt":0,"AltIsGNSS":false,"NIC":0,"NACp":0,"Track":0,"Speed":0,"Speed_valid":false,"Vvel":0,"Timestamp":"2016-02-24T04:05:28.367Z","Age":1.25,"Last_seen":"0001-01-01T00:00:17.85Z","Last_alt":"0001-01-01T00:00:00Z","Last_GnssDiff":"0001-01-01T00:00:00Z","Last_GnssDiffAlt":0,"Last_speed":"0001-01-01T00:00:00Z","Last_source":1,"ExtrapolatedPosition":false,"Bearing":0,"Distance":0}

@peepsnet
Copy link
Contributor

Is #241 addressed in this update?

@ghost
Copy link
Author

ghost commented Feb 25, 2016

@peepsnet -- it should be. The "Last_seen" variable will be set only when a valid position fix is parsed; it is currently set on any valid speed, altitude, etc. message. After six seconds, we stop sending invalid fix targets; after 60 seconds they're removed from the database entirely.

@ghost
Copy link
Author

ghost commented Feb 25, 2016

Some of the "to-do" items listed above have been incorporated into a new development branch. I'll pull these over into new_dump1090 (and therefore this PR) once I get a little testing.

https://github.com/AvSquirrel/stratux/compare/new_dump1090...AvSquirrel:new_dump1090-mutability

Improvements:

  • Traffic UI updates to show TIS-B / ADS-R status for both 1090 and UAT
  • Improvements to signal strength reporting. 1090ES traffic, UAT traffic, and UAT ground links all now report received power, in dB. Scaling is 0=nominal max of the receiver, with a theoretical lower limit of detection at -48 dB. Realistically, I think signals will have to be above -40 dB to be seen.
  • Applied Stratux-specific messages into dump1090-mutability

Towers page refreshes more quickly, and shows current, average, and peak power (dB) levels. (Old traffic page shows strength as sum of amplitude.)

image

ADS-B targets are dark blue / dark tan with airplane icons. ADS-R targets are also airplane icons, but with lighter colors to show that they're relayed by ground transmitters. TIS-B targets shown the same colors, and replacing the airplane with a satellite dish.

image

@Ergonomicmike
Copy link

All very cool!

I gotta run to meet someone at the airport. So I don't have the time right now to look at your changes.

Quick question: Is the 0 to -48 dB hard coded? Or does it pivot off of tgain? I ask because I lowered the gain of my SDR's yesterday by 6 dB (to 420) and am liking what I see in flight. (Anecdotally, the SDR's seem to work better.) I'm not asking for you to accommodate outliers. Am just wondering if anything will break when some users tweak tgain from its default? (I can live with signal strength being incorrect in my case.)

@Axtel4
Copy link

Axtel4 commented Feb 25, 2016

How does a meer mortal go about pulling this into their build?

@ghost
Copy link
Author

ghost commented Feb 25, 2016

@Axtel4 -- If you just want to make a build for testing, it would be cleanest to clone it into a new project.

sudo su
cd ~
git clone https://github.com/AvSquirrel/stratux.git stratux-avsquirrel
cd stratux-avsquirrel

This creates a new folder ("stratux-avsquirrel") and pulls a copy of my master branch.

One you have a local copy of my repo, you'll need to run git fetch to update your local repo from github (this will let you checkout branches), then git checkout to pull my development branch into your filesystem. Then you can make / make install.

If you run the following commands once you're in "stratux-avsquirrel", it will checkout, make, and install the most recent commit version of my "new_dump1090-mutability" branch.

git fetch
git checkout new_dump1090-mutability
make
make install

Or, if you want to make a specific commit, you can use that instead of the branch name.

git fetch
git checkout 5d4284b
make
make install

@ghost
Copy link
Author

ghost commented Feb 25, 2016

@Ergonomicmike -- the -48 to 0 dB is based on the signal level seen by the ADC stage in the radios. If they go full scale, you'll see ~0 dB, regardless of what you're doing with gain or preamplifiers. If they don't saturate, they'll be somewhat under 0 dB.

Our SDRs report the i and q components of the signal. These are both 8-bit signed integers, so each component is ±127.5 units of raw amplitude at 1-unit resolution. This makes the smallest detectable value = 0.5. Since dB of amplitude = 20 * log(a/a_ref), 20 * log(127.5/0.5) = 48.1 dB. We see the same number if I carry through the power calculations.

For each sample, amplitude of the signal is sqrt(i^2 + q^2). In dump978, this is scaled by a factor of 8 to make full-scale = 1000 units, and then this averaged across all of the samples in a single UAT message. (That's the raw "ss=123" value you see in the uat.log files.) These are phase-modulated signals, so amplitude should remain constant and less than or equal to 1000 throughout the measurement cycle. (Mathematically, the magnitude could be as high as 1414 if the ADC saturates and Q & I both measure full-scale; in practice, I've noticed amplitudes of 1100-1200).

Dump1090-mutability goes further: it squares the amplitude of each sample to determine power. The power values are then averaged over the message, then normalized such that max power = 1. Fractional power is then reported in the new dump1090 messages. In traffic.go, I take the fractional powers, and calculate dB = 20*log(power).

So, to get the towers and UAT traffic to match what dump1090 is doing, I just have to rescale the max value from 1000 to 1. The code I committed last night converts to power first, but in retrospect, this isn't really needed for single samples if I'm reporting dB since by definition 10 * log(amplitude) == 20 * log(amplitude^2).

@Axtel4
Copy link

Axtel4 commented Feb 26, 2016

@AvSquirrel . Thanks. I built the code this evening and I have been doing some testing. At 1090 the received power is between -35 and -25 dB. (I don't have enough UAT traffic here to do much testing.) I find it interesting is strongest signal is only -25 when the aircraft is in close proximity. When the targets drop below -35 dB they drop off the list. Note: I am still running close to full gain on the 1090 dongle.

Inserting a 3 dB pad in the antenna line reduces the signal by approximately 3 dB. Which tells me I am not saturated and the scaling looks about right.

I took a look at the ES log and wow there is a lot of good info in there now. But, I found out I don't want to let it run long as it gets huge when it is uncompressed (200+ MB for 30 minutes of data.)

How does "SignalLevel:0.000844" correlated to the Signal Level on the traffic page?

Is there a way to split these back out into their own columns without the colons? It adds a bunch to extra steps to remove the : before processing the data in Excel for analysis.

Lat:39.947732
Lng:-95.125122
Alt:43000

Also, can the ICAO address be converted back to Hex instead of being presented in decimal?

"Icao_addr":10504163 = 0xA047E3
I could convert it back to Hex in Excel but the Mode S address just looks strange in Decimal.

@Axtel4
Copy link

Axtel4 commented Feb 26, 2016

@AvSquirrel

Following up on:

How does "SignalLevel:0.000844" correlated to the Signal Level on the traffic page?

Reading above I see where you wrote,

Fractional power is then reported in the new dump1090 messages. In traffic.go, I take the fractional powers, and calculate dB = 20*log(power).

So if the fractional power is 0.000844 then dB would be:

dB = 20log(0.000844)
dB = 20 (-3.07)
dB = -64.4

But I didn't see any values that low on the traffic page.

Edit:
10log(-3.07)= -30.7 dB is closer to the values I saw on the Traffic page.

cyoung added a commit that referenced this pull request Feb 26, 2016
Dump1090 improvements. Overhauled traffic parsing. UAT groundspeed and air/ground status bug fixes.
@cyoung cyoung merged commit 7007d18 into cyoung:master Feb 26, 2016
@Axtel4
Copy link

Axtel4 commented Feb 26, 2016

I drove by it local tower this morning. Within a eighth mile the tower signal peaked at +1.46 dB. The lowest signal I saw was -29.7 dB. At -29.7 dB the Tower page would stop updating then it would go blank. I thought maybe I would see more than a 30 dB delta in the received signal strength since the Dynamic Range of the dongles are around 48 dB.

@ghost
Copy link
Author

ghost commented Feb 26, 2016

@Axtel4 --

Dump978 reports normalized average amplitude (voltage) for the message, and dump1090-mutability reports normalized average power. Conversions that take into account the scaling factor between the two are therefore:

dump978_amplitude = sqrt(dump1090_power) * 1000
dump1090_power = (dump978_amplitude / 1000)^2

As you seem to have found, the conversion to dB takes this into account: dB_p = 10_log(power), but dB_v = 20_log(voltage). So, using dump978 amplitude of 50 as a reliable 978 MHz message reception threshold:

p_dump1090 = (50 / 1000) ^2 = 0.0025
dB_978 = 20 * log (50/1000) = -26.0 dB
dB_1090 = 10 * log (0.0025) =  - 26.0 dB

On 1090 traffic, I'm typically seeing position fixes starting at about -30 dB, and they're reliable at and above -25 dB. UAT amplitudes of 50 also corresponds nicely to this -25 dB threshold. Altitude messages are shorter, and I'll see them down to about -35 dB.

I thought maybe I would see more than a 30 dB delta in the received signal strength since the Dynamic Range of the dongles are around 48 dB.

Keep in mind that 18 dB is only three doublings of signal strength above the detection floor. I'm still astounded what we can do with these little $20 radios.

image

@Axtel4
Copy link

Axtel4 commented Feb 27, 2016

@AvSquirrel

Thanks. That helps a lot. As you I am still amazed at the capabilities of these little dongles.

This is the a plot of the tower signal strength I recorded this morning while driving into work. I mentioned this in the previous post. My reception starts about -30 dB and peaks around +1.4 dB.

ixd_tower_26feb2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants