Position Hold and Navigation issue

[giacomo892]

Hi. I've already wrote about this issue in the forum and no, unluckly, I do not have a blackbox log but a video. I'm posting here just because a better analysis could maybe fix the issue and cause i've found a previous similar issue that is now declared as closed. (#431)

First of all. Setup.

1.6.1 NAZE. F450 QUADX. UBLOX M8N,HMC5883,BMP280,MPU6050.

And the conf diff:

feature GPS
feature TELEMETRY
feature PWM_OUTPUT_ENABLE
serial 0 17 57600 38400 57600 115200
serial 1 2 57600 38400 0 115200
aux 0 1 0 950 2050
aux 1 1 1 950 2050
aux 2 3 0 1600 2025
aux 3 3 1 1600 2025
aux 4 9 1 1600 2025
aux 5 8 3 1650 2050
aux 6 19 2 1850 2050
set acc_hardware = MPU6050
set acczero_x = 164
set acczero_y = 16
set acczero_z = -12
set accgain_x = 4071
set accgain_y = 4082
set accgain_z = 4012
set align_mag = CW270FLIP
set mag_hardware = HMC5883
set mag_declination = 230
set magzero_x = 55
set magzero_y = -133
set magzero_z = -33
set mag_hold_rate_limit = 45
set baro_hardware = BMP280
set rx_min_usec = 989
set rx_max_usec = 2100
set failsafe_procedure = RTH
set vbat_scale = 50
set nav_max_speed = 700
set nav_max_climb_rate = 400
set nav_manual_speed = 500
set nav_manual_climb_rate = 300
set nav_mc_hover_thr = 1575
profile 1

set mc_p_pitch = 79
set mc_d_pitch = 45
set mc_p_roll = 79
set mc_d_roll = 45
set mc_p_yaw = 80
set mc_i_yaw = 40
set max_angle_inclination_rll = 350
set max_angle_inclination_pit = 350
set rate_accel_limit_roll_pitch = 4000
set rate_accel_limit_yaw = 1800
set roll_rate = 35
set pitch_rate = 35

Basically I had some other times when the quad was in position hold and magically started to drift away (fast!) like @xdigyx reported in #431 "Today I had a situation when at pos hold mode my hexacopter rapidly started to fly away. "

So maybe the fact that this time happened during a mission (and at the final stage = position hold) urged me to post this issue since there maybe something left from that issue.

I've checked for I2C errors and GPS errors after landing and they were at zero, so I assume all the sensors were working.

More, I did another flight without rebooting the FC also in position hold, even fully yawing the multi to check for mag issues, and it held correctly without toilet bowling. (an issue I never had before to be clear)

Here is the video on what happened https://www.youtube.com/watch?v=_kv2N9qdhfc

I've annotated the video , please check it using desktop version of youtube.

[stronnag]

I'm glad you mentioned #431, as i's a canonical example of how to diagnose, fix and demonstrate as fixed such problems. In particular:

• Unusual occurrence on a machine known to fly iNav successfully over 9 months prior
• Logging of every instance
• Rigour and repeatable set of test cases

All the above contributing to a discussion that led to a collaborative and successful analysis as a direct result of rigorous log collection and repeatable test cases facilitating the elimination of false trails before we reached the correct outcome after, iirc, 23 days --- probably the longest continual bug hunt we've had.
#431 was not just "declared closed" it was closed because evidence was presented to demonstrate that a software fix had corrected a bug.

Bottom line: We need evidence.

[Redshifft]

Exact same experience with an Afromini on older firmware 1.3 and ublox 7 - this was a test bed and was perfect for a few weeks when built. After a 4 week layup it started exhibiting this fault again no logs but I will say the movements are quite fast and can be controlled (fast as in quicker than nav speed)

I would often test machines by physically dragging out of position hold and the machine would fly back to position much slower than exhibited above so that would indicate to me something more that intermittent gps data, it is certainly not vibration induced for me anyway,

[giacomo892]

On the same exact day (20 minutes before) with another battery I had something like >10 minutes continous position hold with a nice breeze and the quad was solid at it's place.
I haven't yet found an external condition that let the issue begin.

[digitalentity]

My best guess without logs is GPS hardware or coverage issue.

[digitalentity]

@Redshifft @giacomo892 please record logs of your issues - troubleshooting by video is next to impossible.

[Redshifft]

@digitalentity , @giacomo892 asked me to contribute to this issue, we all have the lack of ports issue for logging i'm assuming.
If it manifests after a minute or 2 in PH it should not "move fast" unless it's glitching out of PH ? PH is well filtered and slow to do anything - you know the code, how long will PH wait without valid (say GPS) before it flys away at speed to some location ? I say it should never and just rely on inertial sensors ?

The test would be to let it go and see where it stops (if it stops) :) like I say anything more than small position changes need to be filtered in PH - the mystery is why this fault only started happening and never a hint of the problem before, and I test stuff like you would not believe ;)

@giacomo892 Did yours just start doing this having been ok previously

[giacomo892]

I always had this issue since I'm using INAV. (I started since 1.5.1). I also do a lot of tests. Even tested the GPS alone on the bench leaving it on for many hours and it seemed to work.

I tried to leave it alone during the issue and you can see result in the video. At first it seemed like a normal behaviour on WP overshot (I never seen an overshoot before to be clear) but then it started to go around the final WP point at speed. Even worse when I switched back to position hold, at that point it started to fly away at navigation speed.

@digitalentity I've ordered and SPI microsd slot 4 months ago... haven't yet received it... If I receive it I can solder a openlog blackbox.

It's strage issue because if you disable pos hold and you enable it again it works as far I can tell.

The very first time I had this issue on a very windy day and I tough my quad could hold the wind. It started to fly away on position hold. I panicked and disabled it and recovered manually the quad.

About the coverage: The very first happened in a place were I had "only" 12 satellites but now happened with 18-19 in view without any obstacle as you can see from the video. There aren't pratically any interferences in that place. No power wires, no base stations and so on.

It could be due GPS outputting bad positions but as I said there aren't logged GPS errors and position hold works if you immediately disable and enable it again.

I could try maybe a self compiled version with glitch detection enabled that increments GPS error count in case of a glitch so I can easily monitor it from ezgui without the needing of a blackbox.

But @digitalentity already told me that glitch detection isn't very tuned. If you can provide better glitch radius and acc parameters I can try it in order to check is the GPS is sending crazy positions.

[Feldsalat]

Similar problem with my quad. Displayed speed in OSD disappears for 6 or 7 video frames then a coordinate jump around 30m and speed 246km/h. Inav responds with very fast correction in poshold. Happend after 9 minutes flighttime with Inav 1.6.1
No log, because older F1 board :-(
GPS: Beitian 880 FC: Flip32 without Mag
I think it is perhaps a hardware problem but why this is not noticed by glitch detection?

[giacomo892]

Could maybe switching to INAV computed GPS speed (from lat and lon change) instead of using the GPS module one fix this issue?

[Redshifft]

@Feldsalat yes as your osd display a high speed departure from PH.

It would seem (GPS) fault that triggers the wild reaction - If the GPS only sent good solid data (which is unlikely) we possibly would not see this issue but there is more to it as fortunately my Micro iNAV quad (Afromini F1 ublox7) does not exhibit this problem, and this has had lots of Flying. The difference is it has "Stronnag special" fw (stripped out with Blheli passthrough and Frsky telemetry added) Also a better Power supply for the board with a nice capacitor across the regulator.
I could try that fw on the test quad I suppose ;)
I would in the meantime recommend people adding at least a 330uf cap on the boards 5v rail/earth to protect the thing from back emf from the motors/esc's

[giacomo892]

I've enabled glitch detection logic using built-in parameters by decomenting line 638 in navigation_pos_estimator.c

#if defined(NAV_GPS_GLITCH_DETECTION)
    isGPSValid = isGPSValid && !posEstimator.gps.glitchDetected;
#endif

and also adding gpsStats.errors++ around line 382. This should let us easily without blackbox if glitch detection logic fired up by simply checking GPS errors with phone/configurator after landing.

if (detectGPSGlitch(currentTimeUs)) {
                    posEstimator.gps.glitchRecovery = false;
                    posEstimator.gps.glitchDetected = true;
                    
                    //add an error if glitch is detected
                    gpsStats.errors++;
                    
                }

I've build the firmware using gcc-arm-none-eabi-6_2-2016q4. I hope this is the correct compiler.
I flashed it. (https://dl.dropboxusercontent.com/u/18187764/inav%20custom.jpg)

GPS works but I cannot fly until this weekend.

If someone want to try it I think it will be totally fine since those are the only modifications.

Download Link:
https://dl.dropboxusercontent.com/u/18187764/inav_1.6.1_NAZE_GLITCH.hex

Edit:

I left the board near a windows for some hours and glitch detection counter has increased: (i'm using an ublox 7M now)

https://dl.dropboxusercontent.com/u/18187764/gps%20errors.png

[WaspFPV]

I had the same behaviour in the past and tracked it down to a malfunctioning compass. It did about the same as in the video, first passing waypoint, spiral around it, then flying away.

[giacomo892]

I'll check the compass from sensors tab when I land if the problem happens again.

[stronnag]

Probably not going to help if it's a transient issue caused by power related mag interference. Hence the frequent requests for a bb log.

[giacomo892]

Yes, you are right. But if the mag issue is transient it should not trash the whole PH.
I'll report also with the new firmware that i've compiled.

[Redshifft]

Right, I have just put some fw on the problem test Quad that has been no problem on something else. I might be able to test tonight if it does not rain.
@giacomo892 I still have to wonder why the Copter moves so fast when it's in PH and this issue manifests..it is just illogical to me !!

[giacomo892]

@Redshifft have you had time to fly my firmware?

That's the real question. Maybe it receives wrong coordinates or speed from the GPS module.

[Redshifft]

Ok, had about 6 flights 90% in PH and a couple of RTH, the wind and gusts were about the limit for iNAV and the 250 Quad, no problems encountered - Hardware was not touched in any shape or form from before and original configuration just copied across, down to you software boys to figure out now ;)

[giacomo892]

@Redshifft pardon, what software modification have you done ?

[Redshifft]

I said previously, my mate "software" compiled me a stripped out build.

[giacomo892]

So without any code modifications? Just stripped out things? Can you list them please.

[Redshifft]

Stronnag built it, I asked everything be pulled out and Frsky telemetry and Blheli passthrough added

fw attached rename it to rar ;)

[giacomo892]

I've crashed my F450.
Luckily I was on grass and nothing damaged. It did a roll in the air and started falling.

[Redshifft]

@giacomo892 What does that mean ? what fw ?

[giacomo892]

I have no idea why it crashed. Some minutes before I did lot of yawing to check if mag was working and the position hold was working.
Then suddenly it flipped and crashed. I was luckly at 10 meters. I was running my build 1.6.1 with glitch detection. I saw glitch detection working at least one time but the quad tried to escape like usual and at least two times but I stopped it forcing it in the opposite direction with the remote control

[Redshifft]

Flip = motor
Put my fw on and test as yours in consistently showing this problem

[giacomo892]

I've tried to takeoff immediately after the crash and the copter stays in the air. I'll try yours soon.

[giacomo892]

@Redshifft I've flashed your firmware. If the weather allows i'll try to fly it today. Have you tried mine if you still have PH issues?
@digitalentity Regarding my mid air flip and crash. Is there something related the mag that can screw up the IMU and crash in ANGLE mode? The motors seems to run fine.

Do you all have experience of esc+motor that fails randomly in air but works right after?

I've HobbyWing Xrotor 20A ESC. Should I put BLHELI on them?

[digitalentity]

@giacomo892 mag data can't screw roll/pitch data and cause a flip. This is most likely hardware issue caused by one of the following:

1. Compass chip messing up with I2C wiring and causing gyro hickup - which usually results in imminent crash.
2. Signal issues with one of the ESCs, usually caused by no signal ground from FC to ESC.

[Redshifft]

@giacomo892 The weather has turned to crap for a few days here :(
More important you test my fw which has not displayed problems on 2 Quads.
Do not change anything on your quad, sure it's unfortunate if you have other problems but we have to back-back the fw.

@digitalentity can you Just confirm in PH the max nav speed in all directions?

I have some ideas but await @giacomo892 results first.

[giacomo892]

So, just to recap, we have now a "randomly" stripped down INAV 1.3 firmware which is bug free.
Nice situation to start the bug search. I thought that the "modified" 1.3 had some fixes in it.

[stronnag]

The only changes, afaik in a 1.4 pre-release build was to remove some standard features and add in some non-standard. No other code changes. Just a placebo.

[giacomo892]

Maybe I should try building 1.6.1 with the same feature set and try to fly it.

It should have:

1. No Blackbox
2. FrSky telemetry (should be already enabled, which I do not use)
3. BlHeli Passtrough (I cannot find the 1Wire #define to enable it, can you point it out please)
4. Only MPU6050,BPM280 and HMC5883 selected,
5. Disabled SPI

Correct me if the feature set should be different.

[Redshifft]

What's all this random and placebo stuff ? :D

[stronnag]

I can check the features later.

[WaspFPV]

@giacomo892 main/src/target/Naze/target.h, near end of file:
//#define USE_SERIAL_4WAY_BLHELI_INTERFACE
By the way, do you really need it? I just save my config dump and switch to Betaflight.

[giacomo892]

andre92 Thanks. I use it sometimes to check ESCs but I want to include it just to have the most similar firmware to the modified 1.3
Betaflight is intended for racers and it wont provide any GPS features which are being discussed here.
And please fly the custom INAV 1.3 first and report if RTH and POS HOLD works

[giacomo892]

I've compiled the firmware as I said earlier.

I've used gcc-arm-none-eabi-6_2-2016q4 :

arm-none-eabi-gcc --version
arm-none-eabi-gcc (GNU Tools for ARM Embedded Processors) 6.2.1 20161205 (release) [ARM/embedded-6-branch revision 243739]

inav_1.6.1_NAZE_minimal.hex.zip

The firmware size went from default 127676 bytes to 118400 bytes.

Here are target and common modified and commented files.

target.h.zip
common.h.zip

I'm unable to fly till the next weekend. So if someone can try this before me it will be awesome.

[Redshifft]

Did you do as you said ?
"No Blackbox
FrSky telemetry (should be already enabled, which I do not use)
BlHeli Passtrough (I cannot find the 1Wire #define to enable it, can you point it out please)
Only MPU6050,BPM280 and HMC5883 selected,
Disabled SPI"
I would have asked everything that I did not need driver wise be pulled, don't forget UBLOX for the GPS.

[giacomo892]

@Redshifft I've attached common.h and target.h if you want to check.
UBLOX is enabled too since it is already enabled in the official build.

[Redshifft]

LOL, I did what's all this Mac stuff :)

[giacomo892]

Holy cow. Compression Utility leaves some files inside zip :) Ignore them

[stronnag]

I flew F1 (FLIP32) 1.6.3 on my half rebuilt 420mm quad. Somewhat unsatisfactory as it was completely untuned (just copied a CLI dump from a 250 onto it), and due to the rebuild, the mag was closer to the power than I'd like.

Regardless it works (for me). After about 12 minutes of entirely OK POSHOLD (no flyaways, no crashes), bit bouncy due to the lack of tune, but in the same place POSHOLD, I went to do something less tedious.

[giacomo892]

HI all.
Just to keep you posted about what going on.

I'm building a DIY serial openlog blackbox so I can connect it to the official 1.6.1 and let you all see what is going on.

I hope it will work ok and so I will finally post some logs here. Even if I don't get a flyaway we should see strange things happening when in POS HOLD.

I just miss a level converter between SD and Arduino and I should ready to go.

On sunday there will be rain here, so I hope I'll have time to set it up for saturday.

Beside that, anyone has flown the stripped 1.6.1?

@stronnag I'm glad you don't have the issue but trust me, it's somewhere hiding there.

EDIT: Perfect! My soldering iron broke down. I hope I can find a new one today.

[giacomo892]

Yay! I've flown with the blackbox! i've setted 1/32 logging ratio and using cleanflight explorer I still have some corruption?
Do I have to use cleanflight explorer?

Or maybe the SD CARD is not fast enough. I'm using 250000 as baudrate so that should not be a problem.

I had no flyaway today and all worked really well.

Can you take a look in the logs and say me if they are ok or not?

Thanks

[stronnag]

I replayed the logs in mwp, looks normal (just like I'd expect for iNav on a supported FC); at least for the amount of time I'm prepared to watch a replay of PH. So what's all this fuss about?

btw. You do know the two LOG files are identical?

btw, btw: You should use the iNav Explorer (or mwp for pretty geospatial display)

[giacomo892]

Ok. Blackbox explorer show weird data.
Do you think this rate is ok for debugging navigation or I higher it a bit?

[stronnag]

Rate is OK for general navigation debug, if you can identify a specific problem, then a higher rate would help.

[giacomo892]

Are they 100% identical? If so it could be an issue with upload

If the log is ok I double the rate. It should not be a problem.

[stronnag]

100% the same.

I didn't look at the whole log, but random bits looked ok in INav Explorer

[giacomo892]

Really strange since the file size are different. How can they be identical?

[stronnag]

Seems I can download the same file under two names. Strange indeed.

[stronnag]

btw: https://github.com/iNavFlight/blackbox-log-viewer

We need to update some links ...

[giacomo892]

Sorry for "spamming " the issue .
I've tried the INAV explorer manually loading the app in chrome and it works. Is there any possibility that the app will be published in the app store?

Btw. Yesterday​ I did almost 25 minutes of GPS holding including moving in GPS cruise. No flyaways.

Now that I have a blackbox ( I've configured it to log 1/2) and it just drops some loops every now and then I'm ready to chase the bug more than ever.
It's rather strange since before installing the blackbox the issue was always present :(

[giacomo892]

I recorded a GPS glitch on blackbox on the 5th flight.

You can notice the position jumps many meters far away from the quadcopter after it took off and other many succesful flights in the past weeks.

While the quad was still glitching and GPS position was converging to the right position I enabled GPS HOLD and visually it felt very unsure on holding position but without any weird accelerations and fly aways.

After some seconds I disabled the HOLD and renabled it again without any flyaways.

After landing I tried another flights and they worked well as you can see.

LOG00275.TXT

Now imagine what can happen if the position jumps while the GPS HOLD is on.

Can you get some useful info from the log to prevent this?

EDIT: It glitched also on the first part of the 6th flight but the position quickly converged to the correct one and the it flew well.

Viewing this log can be nice to see:
1)position hold logic while position is wrong (converging to the correct point)
2)simulate with real data what the position hold controller would have done if it was enabled during the glitch
3)check if glitch detection would have helped.

[stale]

This issue / pull request has been automatically marked as stale because it has not had any activity in 60 days. The resources of the INAV team are limited, and so we are asking for your help.
This issue / pull request will be closed if no further activity occurs within two weeks.

[stale]

Automatically closing as inactive.