Lost ground link multiple times in v0.7b2 #267
Comments
|
I also observed this on .7b1 last weekend. From: GrayFlyer [mailto:notifications@github.com] I flew v0.7b2 today with iFly on my '720 and a Android device. In less than one hour I lost ground reception at least 10 times on both devices while receiving 4 to 6 towers. It always restored connection without any intervention on my behalf in 30 to 180 seconds. Everything on the Web UI looked fine. Everything on the iFly connected devices page looked fine except for "ground link status" — |
|
What indicators were showing you that you "lost ground reception"? |
|
WX reception would disappear (I normally fly with FF’s Categories selected) and the indicator of how many towers were being received (top left of the FF map) went away, then came back red – showing 0 towers, then shortly showed a larger number and operation was restored. I did see on the stratux webUI that I had towers. They just disappeared from FF. From: cyoung [mailto:notifications@github.com] What indicators were showing you that you "lost ground reception"? — |
|
iFly has a connected devices page: |
|
I'll give it a try with iFly on Android. Is it the same binary you use for your Fire? |
|
|
I plan to go flying in about six hours. I'll pull the latest 23 commits. Aside from the usual logs, is there any other testing that I can do to help isolate what's causing this problem? iFly creates a adsbraw.bin file. But I expect that that is proprietary. (Although if I send them that file, they might be able to tell me why they report no ground connection.) |
|
I flew v0.7b2 + 23 commits with iFly v9.4rc1 today. Flew two half hour legs with a one hour stop over. I left the Pi on the whole time and i left my tablet on (although in sleep during the stop over.) I toggled back and forth between the Pi status web UI and iFly in flight to exercise things. (Only one tablet in flight today tho.) The Stratux worked perfectly. Absolutely no problems in flight. No tower drop outs, wx coming in fine, etc. It wasn't until the end of the flight, when I was getting ready to shutdown the Pi with Raspi Check that I noticed the blue light on the Edimax wasn't flashing. I tried Axtel4's trick of shutting off the Wi-Fi on my tablet. But that didn't work. I would post the stratux.log and the syslog. Unfortunately, they both appear to be corrupted. "Does not look like a text file" is the message in Linux. (I had deleted the old stratux.log before the flight to keep things short.) I'm guessing that I had some kind of catastrophic failure that corrupted these files? I seem to be the only getting the blue light out problem. So I'm going to assume for now that this is an Edimax hardware problem. I am planning to borrow a neighbor's Edimax to test. Could it be a bad SD card? I should swap that out too. |
|
P.S. The guy flying with me today brought his Stratus v2 along. I was going to compare tower reception. But as I was getting 5 towers today in the Phx area, I totally forgot. And too, every target his FF showed with his Stratus, my iFly showed with my Stratux. |
Should be able to download it anyways, try using the web interface. http://192.168.10.1/logs/stratux.log. |
|
Battery too low for the stratux at the end? On 02/19/2016 07:41 PM, Ergonomicmike wrote:
|
|
You might want to run an fsck first, Mike. On 02/19/2016 09:16 PM, cyoung wrote:
|
|
@cyoung Thanks. You're correct. The log appears in the Stratux web UI. However, it's not like the other logs that I can download. Best I can think of doing now is cut and pasting. But now that the log is showing, I'll copy the file in Linux to a flash drive and upload it on the web. (Update: Am using Puppy Linux and it shows syslog.1 just fine. In the past, it has shown stratux.log just fine too.) @skypuppy On fsck - I forget - does the Stratux already do that automatically on every boot? Or when a dirty flag is set? (Until RO comes to Stratux.) As for the low battery - I've had excellent service with the Anker. It normally goes for 10 hours. It only had 2 on it today and is still showing 3 bars. |
|
stratux.log and syslog zipped and posted at Found that they open in a web browser. (Geany and "open as text" barfed.) Looks like the problem occurred at 23:17Z (Feb 19). I'll be interested to hear what you all say happened. Update: or 23:10Z, depending on which log you're looking at. |
|
Mike, Makes me wish the RPi2 had "multi-boot" capability, where I could let it Skypuppy On 02/19/2016 09:36 PM, Ergonomicmike wrote:
|
|
If I EVER get an IDE working with cross-compiles, I intend to make an Skypuppy On 02/19/2016 09:58 PM, Ergonomicmike wrote:
|
|
Ran fsck -f -v on both partitions. The second partition had some problems with inodes and fsck fixed them. Whether the fsck problems were the cause of my Edimax problem or a result, I do not know. |
|
I'm having trouble finding clues of some abnormality at ~2016/02/19 23:17, what tipped you off that it happened around this time? Was it just around when you noted that it happened? |
|
@Ergonomicmike , @cyoung -- look at syslog starting at 23:10:19Z. I think your USB controller had a meltdown. All of the USB devices in stratux.log also shut down. |
|
Ah, so 23:10Z per my update above. (Sorry, Chris, for the initial false lead.) Okay, USB controller = hardware, right? So hardware failure means I need to buy another Pi? Or - could one of my USB devices be causing the controller to fail? (Probably my first SDR, since I think that I've been having this meltdown from the beginning.) It's clearly intermittent. Would these logs catch that failure or are the logs not granular enough for that detail? And thanks to both of you for looking into this. (Update: And good to know it's not a software problem.) |
|
I don't think it is failed hardware. raspberrypi/linux#35 -- they theorize that it is not power related, no resolution. http://www.netnode.de/howto/USB_modem_power_issues.html -- power related. Anyone know of a good add-on to the RPi to start logging voltage? There's this: http://amzn.com/B00ORN78WE, but it doesn't help much with intermittent problems. Perhaps a low voltage cut-off so that it's obvious when it happens? |
|
this is how I would diagnose it, Mike. I would check power issues first. On 02/20/2016 12:08 AM, Ergonomicmike wrote:
|
|
Be better if it had logging capabilities. Hmm, I remember something from Sparkfun or Adafruit that watches power if it is temperature related, and so on. On 02/20/2016 12:38 AM, cyoung wrote:
|
|
I will put the Edimax on a short USB cable in case it's an EMI problem. |
|
I remember reporting blue light out when coming back from Big Bear at night in early Feb at 9500'. It was cold up there. That would be a point in favor of not heat related. |
|
I see the UAT shutdown reports in the stratus.log. Is there a way that the software can either restart the controller or command a self-reboot when it sees that report? The latter isn't elegant, but at least would recover from the failure. |
|
Can we call this issue "The Blue Light of Death"? (Parody-ing the MS BSOD.) |
|
We should look into reinitializing the wifi dongle when disconnected, as
|
|
I looked at AvSquirrel's analysis of syslog for the case of the Edimax physically disconnected. There is a "USB diconnect" message - which also appears in syslog when the controller melts down. So I presume that one code trigger off that message to try to reconnect in both cases. Now, naturally, I'm all for addressing Issue 246. But in the case of a melt down, we don't know if the controller will accept commands (to reconnect), or if it's in some indeterminate (software?) state. If in an indeterminent state, then wouldn't a reboot be necessary? But if some transistor or gate in the hardware is stuck on a rail, then it might be that not even a reboot would fix it. (In which case, ironically, the Anker's foible that causes a restart after shutdown would come in handy.) |
|
So I'm looking at this solution that Chris found. My first thought was to run some 22 gauge wire from the pads near the Pi's micro-USB to the 5v and gnd of the USB connectors. But then I remembered seeing a line in the ccon.txt about increasing current to the USB. So does that mean that the Pi is somehow involved in regulating power to the ports? (Why?) If so, then I ought to Xacto the power run(s) to the ports and just wire them directly to 5v in at the micro-USB. Please, before I do this, would someone tell me why this is a bad idea? Edited to correct name of config file. |
|
USB provides IIRC 100ma to the ports for detection of insertion. At that point the device tells the port (electrically) how much current it wants, up to a limit for each type (1.0, 1.1, 2.3, 3.0) and the port can switch to that. I'm not sure what the .conf is doing, but it is risky to ask a controller to overdrive the port. The transistors that drive power have some overhead room, but it's not always a good idea to ask for more. Think of doing so as overclocking. For parts designed to do that, it's ok. For others, there's the risk of damaging the output driver. The solution of providing additional power (in parallel) from another USB port or 5v supply is a better choice, but be wary of simply asking too much from the pi. If you remove the USB port on the Pi from realizing currents being pulled through it, you're likely to cause the part to not work right. Josef
|
|
"2. Any data bursting on any port (especially SD writes) spikes internal buses - you aren't going to see that with a typical multimeter." Comment as of Feb 18, 2016 at https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=135214&sid=af90061d8268c1b4ebeff6788148f497 I don't know what writes the Stratux is bursting. But if true, and if the blue light out is power related, then going to RO might mitigate the problem. |
|
@jzeevi The link here http://hackaday.com/2015/04/06/more-power-for-raspberry-pi-usb-ports/ explains what the config file is doing. (Switches a resistor on or off that controls a current limiter.) A commenter has an even cleverer way to change only one resistor so that the soft switch in config.txt still works. |
|
4.76 V at the USB w/ fully charged Anker, as others have measured before. Right on the low edge of the USB spec. Man, those surface mount resistors are tiny. I'm gonna have to buy a smaller soldering tip and hope that my eyes & hands are up to the task. |
|
Sometimes the more practical solution is the better one. I soldered a jumper. http://s10.postimg.org/bf2aokkax/5v_jumper.jpg Picked up 100 mV at the USB ports. Up to 4.85 volts now, under load. We plan to fly Wednesday. So hopefully will have a report by then. |
|
Before cutting those itty-bitty traces, I recommend you verify it a Mike, you must have access to some way to measure current in-line, don't Regarding the Anker "3 lights," I have stopped trusting indicators on Skypuppy On 02/20/2016 07:56 PM, Ergonomicmike wrote:
|
|
No, sadly I don't have an easy way to measure current. If this issue persists, I might buy one of those USB measurement tools that Chris linked to. I would probably have to open it up to get a 'scope on the shunt or Hall Effect or whatever they're using to detect current. |
|
Data point: The Edimax seems to be "smart." Plug it into a power pack. The power pack indicator will light up. The Edimax never does. (Nothing to xmit.) So, within a few seconds, the power pack's indicator light will go out. Which means that the Edimax has shut itself down? (Activity checker?) Can the Edimax be restarted from this state? Edited for clarity. |
|
Found a bug in traffic.go. Fix for this will be rolled into PR #285. Lines 296 to 298 in 65559a3 FIS-B queue send interval ( Depending on how quickly iFly is "timing-out" stale uplinks, this may be enough to show you losing towers. |
|
Excellent find! Looking forward to seeing if that solves both the iFly "Connection lost" problem and the BLOD problem. (Possibly a burst of data overwhelming the USB controller?) To be clear, iFly makes a distinction between tower count (which I'm pretty sure they get direct from Stratux) and "Ground Link Status". (Which, IIRC, is a new parameter, that they're inferring from data throughput.) |
|
@AvSquirrel @Ergonomicmike Thanks for everyone's input and help! |
|
@GrayFlyer -- looks like it was around v0.5b2 (e6ad9aa#diff-4ba3b85a06589d714ee8177de6236fd0), but I'm guessing it didn't manifest until message batching in the v0.7 releases allowed the queue to empty. |
|
@GrayFlyer Thanks for correction. I had it backwards. |
|
I flew this morning with v0.7b3 for about 1.5 hours plus the system up during a fuel stop.I had absolutely no problems with wifi connection or loss of ground reception. I had two devices connected; an iFly 720, and a Fire tablet. Unless there is some significant difference between b2 and b3, I'm now wondering if what I saw before with repeated ground reception drops may have been FAA system glitches? Thanks for everyone's input and work! |
|
@GrayFlyer - the issue hadn't been addressed in the code you tested, so it may have just been luck that it worked (unless there's some other code you were testing?). The proposed fix will be in the next release - would be great if you could give it a try when it's out. |
|
@cyoung - No other code involved, no changes to my hardware. I'm looking forward to flying the next release, but it will be after my annual which begins 3/1, before I can. |
|
I think it was related to the network issue mentioned above by @AvSquirrel. That has been addressed, reopen if it happens again. |
I flew v0.7b2 today with iFly on my '720 and a Android device. In less than one hour I lost ground reception at least 10 times on both devices while receiving 4 to 6 towers. It always restored connection without any intervention on my behalf in 30 to 180 seconds. Everything on the Web UI looked fine. Everything on the iFly connected devices page looked fine except for "ground link status"
The text was updated successfully, but these errors were encountered: