[Bug] Gryro Drift Issue During Longer Runs - How To Debug #1687
Comments
laurensvalk
added
software: pybricks-micropython
|
Thanks for your detailed feedback.
You can use the print command for this. You can print multiple columns: For example you could do this in a loop, or as part of the rest of your code. print(timer.time(), hub.imu.heading(), hub.imu.angular_velocity(Axis.Z), sep=",")Then copy the output to Excel to easily make a graph. I would be especially curious if that 15 degrees builds over time, or whether there is a discontinuous jump.
Do you run straight with respect to the start of that maneuver each time, or do you try to maintain an overal compass-like angle?
It's worth trying |
|
Some preliminary observations:
Apart from that we found an external factor: ambient temperature. In hindsight the worst runs seem to have happen either when the ambient temperature was relatively high or wenn there were a lot of runs in a short time (we suspect some heat accumulated inside the hub) or both together. After heating up the entire robot to 45 degrees Celsius (above freezing point) the problem manifested rather quickly and much more severely. Does the IMU have temperature compensation? It seems to be an LsM6DS3(TR-C). I did not find anything hinting to this. However https://community.st.com/t5/mems-sensors/lsm6ds3-temperature-compensation/td-p/294475 seems to confirm the problem. |
Good point!
It can measure temperature, but does not automatically account for it. Potentially, we could do that in our own driver. I've attached the datasheet below in case anyone is looking for some light reading over the holidays 🌞 These are the nominal sensitivities to temperature, for both the rate magnitude and the drift (the "zero rate"). Both will affect the result of a turn.
|
|
On a slightly unrelated note, I think we can also improve the usual calibration on boot a bit. Right now, we wait for stationarity but assume 0 until then. We could potentially save the offset values just like we will be saving a few other IMU calibration settings. Then it will drift a bit less until the first calibration has occured.
|
@DrTom Would you be willing to show the phenomena on a video here or even in pm? |
Yes we can do either. However, it will have to wait until we are back from vacations in about 3 weeks. |
|
I run into a similor issue. I'm using SPIKE Prime hub on a drive base. The run takes about 20secs. After reading this thread, I tried to stop the base for 1sec between steps and print a serise of I also suspect that heat accumulation might be the reason, because I used I will conitnue diagnose and post my finding and update. |
|
Update: |
|
Thanks for your note. Breaking it down into multiple moves does not impact gyro calibration, so this may be masking something else. At the moment, the gyro calibrates mainly the first time it is stationary after boot, and then only gradually updates the calibrated value. Maybe we should measure the temperature and allow for faster re-calibration if the temperature has changed. |
|
Since break down helps, maybe temperature is not the cause. I have also tried shaking and tile the base and table while running, it can't be reproduced. Only turn(-135) in the middle of the run could trigger that. I will try to reproduce with simplest steps and provide a code later. |
|
Does it really produce drift? That is, does hub.imu.heading() give a wrong value and does it drift up or down? Or is the robot not moving by the intended amount? Thanks for looking into this! |
|
Yes, after this -135 turn, it produces a non-zero angular velocity. After the -135 turn when drive base is static, if I print |
|
I just figured out how to trigger drifting and filed an separated bug since I don't know if it's the same issue. See: #1840 |
|
Thanks @scatwang for making a small reproducible case! @DrTom - the likely outcome for now is that the default calibration thresholds are a bit too lax. # Defaults
print(hub.imu.settings())
# You can set stricter thresholds, so it won't easily start to recalibrate while turning steadily.
hub.imu.settings(1.5, 250)The thresholds had been relaxed for the previous firmware release after feedback in #989. Because if the thresholds are too strict it won't calibrate in a competition hall with kids jumping up and down 😄 This will be addressed by making your settings (as above) persist across reboots. So you can choose what works best for you. With that done, I think we can use stricter defaults, so you can relax them in your user script if needed. This is easier for the user than the other way around. So this is quite possibly the same issue, but let's keep both open for now. |
|
If this was indeed the same issue discussed above, then the fix is available for testing using these instructions to try the latest version It would be great to know if this fixes the issue you saw, @DrTom. Thanks in advance for your time. |
|
So, after the team rebuild the robot and reprogrammed everything it seems that the drift problem ist not gone. I can confirm our drift problem with a recent beta build of pybricks. I performed some quick tests today and it seems either related to motion of the hub and/or increase of internal temperature. Some preliminary experiments from today:
For the next tests I plan to recalibrate, then prevent recalibration and exert external temperature; all while the hub ist completely stationary. That should differentiate between temperature or some other cause. |
|
An easy way to change the temperature is to turn all of the light matrix lights on or off. See https://github.com/orgs/pybricks/discussions/675#discussioncomment-3341131 |
We have also seen some small drift <= 2 deg/min; but that isn't much of a problem for a competition run. There are other causes (motion, some hits with elements) which cause the gyro deviate over time by a few degrees. It all depends but you would usually realign a few times (say 2 to 4 times) during the run which will take care of that and some minor drift. However if the drift is in the magnitude of 20 deg/min you would have to realign (and give time to recalibrate) like 8 or 10 times during one run of 2 minutes. Even if that would be possible it would take between 30 to 60 seconds time. |
That is very interesting (also the thread https://github.com/orgs/pybricks/discussions/675#discussioncomment-3341131). I actually thought about the LEDs but it seemed to me that the effect should be little compared to motor controls and battery drain. But maybe it is not. And also I have no idea where the IMU is located within the hub. If it is very close to the LEDs it could easily matter. For the test I plan to use our kitchen convection oven; I assume about 50 or 60 deg Celsius should not cause (much) harm to the hub. |
|
Thanks for the additional input everyone! To help test this, I'll soon add an API option so you can do: In order to get the actual values from the sensor. Then we can better assess what those do, to eliminate builtin offset corrections or calculation errors we might have in the firmware.
I have not yet been able to reproduce this. If you find a reproducible case, that would be super helpful. To double check, can you include the following in your code and share the result? from pybricks import version
print(version)Please also share the output of: @DrTom , do you have any standard LEGO models built, such as the small or big drivebase? It would be super helpful if you are able to reproduce it on one of those, so we can compare. There does seem to be some influence of the LEDS, though not major for the gyro at first glance. With the lights off, the average Z acceleration for my hub over one second is about 9681 mm/s^2. With the light on, it rises to about 9776, which is approximately 1% error. The effect on drift is very small, so doesn't seem to explain the higher than usual drift posted about above. |
This is indeed reproducible by putting it against something like a PC keyboard and then gently rotating it. This is because we use a stationary test where the angular velocity should stay under a threshold for a second in order to re-calibrate. Due to some other recent developments where we will actually save the bias, I think we can choose to reduce the default threshold from 3 deg/s to 1 deg/s. This way it should be less common to accidentally move it slightly while calibrating. Please experiment with the following script if you'd like to see the effect. from pybricks.hubs import PrimeHub
from pybricks.parameters import Color, Axis
from pybricks.tools import wait
hub = PrimeHub()
#
# Default is 3. Reduce to 1 to make it more strict about calibration.
#
hub.imu.settings(angular_velocity_threshold=1)
while True:
hub.light.on(Color.GREEN if hub.imu.stationary() else Color.RED)
print(hub.imu.heading())
wait(100) |
|
Results testing the LED impact on the drift. See the annotated log file and the python program for all details. The hub was placed (inside the robot) on the floor in a room with about 23 deg Celsius temperature. The robot was not touched during the whole process. The room has wood floor glued to massive screed, so no shaking or something which could influence the result. In the first test the hub was turned on, the LEDs were immediately turned off and than turned on after 5 Minutes. The test ran for 10 Minutes. We can see an ever increasing drift up to 13.5 deg/min towards the end of the test. At first glance there seems to be something generally off and if there is an impact of the LEDs it is shadowed by that other thing. After some consideration I suspect that some heat inside the hub builds up until an equilibrium of generation and dissipation is reached. So we repeat this test after the hub has been idle for a while, note that the center LEDs are on during that time. In this second run we can see that the drift "recovers" towards the other direction up to -3.4 deg/min. After 5 minutes, when all the LEDs are turned on, the drift again shifts towards the other direction up to 1.1 deg/min at 9 minutes (the 10 minute mark is not present; a racing condition I took not into account, see the code). We could account for the positive drift by the fact that all LEDs are on now compared to only the 3x3 center LEDs when the hub is idle. For now I would lean towards the following conclusion: drift is sensitive to heat produced by the hub itself; quite significantly so; even the LEDs can have a significant impact. |
|
Thanks for checking! Can you share the script used to produce your results?
While not ideal, this still seems far less than your earlier finding:
Was this using LEDs, or during certain runs? I'll try building up some heat by stalling some motors. |
I know drifting can be triggered by simulating a slow and steady movement. But the issue I mentioned was triggered by running a program. The drivebase was intended to run fast and crash to the mission models. Anyway I will add |
It is attached in my comment above. Somehow I had to add the
The earlier findings were related to an actual run. In this case the robot was sitting still, completely stationary, no movement. For that it seems already significant. This was just to see if the LEDs do play a role. |
It seems that it would possible to read the temperature from the IMU: Would it possible to expose that, too? Also if we could access the drift setting for reading and writing together with the temperature it seems possible to first run some tests reading various values and then set them during a run in a separate tasks. This is probably not a solution for the long run but it could help us getting there. |
We currently do a round-read of the acceleration and gyro registers. We can have a look whether the temperature can be read as part of that. This week I will try to push some of the existing changes so you can individually set and read:
Thanks again for experimenting. It is much appreciated. |
|
Here is a short program that prints the heading every second and the drift since the last second. You can rotate the motor to turn the lights on and off. This has virtually no drift at all for me, whether the light is on or off. from pybricks.hubs import PrimeHub
from pybricks.pupdevices import Motor
from pybricks.parameters import Color, Port
from pybricks.tools import wait
hub = PrimeHub()
dial = Motor(Port.B)
DELAY = 1000
last = hub.imu.heading()
while True:
wait(DELAY)
# Get new heading and drift since last heading.
new = hub.imu.heading()
drift = (new - last) / DELAY * 1000
last = new
print(f"H:{new:7.3f} D:{drift:7.5f}")
# Light on or off depending on motor angle.
hub.display.on(100 if dial.angle() > 0 else 0)
# Green = we are calibrating, red is not.
hub.light.on(Color.GREEN if hub.imu.stationary() else Color.RED)@DrTom - I see you've turned off calibration in your tests. Logging calibrated, time-integrating values while explicitly turning off calibration makes the data a bit difficult to interpret. A drift may seem large, but it could be the result of the not-fully calibrated data, which adds up to big numbers over the course of a few minutes. I suppose it would be more helpful to do experiments with uncalibrated data. When this build finishes building, you will be able to do: data = hub.imu.angular_velocity(calibrated=False)You will find that it is nonzero for stationary. This is what we try to average/estimate during calibration. And then we subtract it during normal use. And we integrate that result over time to produce a heading. So the interesting part would be how fast the above measurement changes over time due to temperature and other factors. All the others (bias, heading) are derived from this. |
After initial calibration: yes. My impression was that this is the point. We let the system do calibration. The drift should be gone in this state (since it is calibrated). The we impose some condition onto the system - like turning on the LEDs. If those produce enough energy such that drift appears this will also be an issue during a run e.g. Since there can not be recalibration during a run we must otherwise compensate for that. So far my naive ideas ;-) At any rate: I just flashed your new build. If repeat my tests and output your data as above it will be useful to you? |
|
Almost, but there is a bit more to it. In the example that the program was immediately started, there would have been only one second worth of calibration. This is very course/rough, more like an initial ball park estimate. It is better than nothing, but it is better to give it more time to obtain a longer average. I think this could explain why you saw bigger drifts in that case. As for experimentation, I'm mainly curious about understanding and reproducing big causes of drift. In this case, I just thought I'd share this early build for your curiousity :) You are welcome to look into it anytime. Otherwise I will get back to it eventually too. If your hypothesis is that the screen is a factor, perhaps you can measure and average the uncalibrated Z angular velocity under different conditions. And perhaps also with a few motors stalling themselves. 3D gyro integration coming up soon! 😄 |
Hm, my hypothesis (for now) is, that temperature is causing drift. I would suspect that the motor controllers contribute much more than the LEDs. The problem testing with motors is that it seems hard to separate contribution from the motion and that from the motor controls (right now I have a build robot I can not take apart). Hence the tests with the LEDs and yet to come external heating. I did not expect to see much impact by the LEDs. But a few Degrees seems already substantial for that.
OK, I see, so I added a full minute for recalibration: async def calibrate_imu():
hub.imu.settings(3, 2500)
while not hub.imu.stationary():
await wait(100)
await wait(1000 * 60) # wait for 1 minute to let the IMU settle moreThe results however don't differ much. It seems more about the initial state how fare we drift to the right or left but the range ist still comparable. Here is the result including your And for completeness here is the full code: from pybricks import version
from pybricks.hubs import InventorHub
from pybricks.parameters import Icon
from pybricks.tools import run_task, wait, StopWatch, multitask
hub = InventorHub()
print(">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>")
print("Testing the stationary drift of the Prime Hub.")
print("Pybricks version:", version)
print("Pybricks IMU settings:", hub.imu.settings())
print("<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<")
stop_watch = StopWatch()
async def calibrate_imu():
hub.imu.settings(3, 2500)
while not hub.imu.stationary():
await wait(100)
await wait(1000 * 60) # wait for 1 minute to let the IMU settle more
async def lock_recalibration():
hub.imu.settings(0, 0)
await wait(1)
async def heading_and_drift_print_loop():
prev_heading = hub.imu.heading()
print("Time[min], Heading[deg], Drift[deg/min], data[?]")
while True:
await wait(60 * 1000)
heading = hub.imu.heading()
drift = (heading - prev_heading)
prev_heading = heading
data = hub.imu.angular_velocity(calibrated=False)
print("{:2d}, {:4.1f}, {:2.1f}, {}".format(
int(stop_watch.time() / (60 * 1000)), heading, drift, data))
async def led_impact_test():
print("### Testing LED impact on drift START: 5 minutes without LED - 5 minutes with LED")
async def led_on_after_for(delay_min=5, for_min=5):
await wait(delay_min * 60 * 1000)
hub.display.icon(Icon.FULL)
await wait(for_min * 60 * 1000)
await wait(3 * 1000) # wait for last output to be printed
stop_watch.reset()
display = hub.display.off()
await calibrate_imu()
await lock_recalibration()
hub.imu.reset_heading(0)
await multitask(
heading_and_drift_print_loop(),
led_on_after_for(5, 5),
race=True),
print("### Testing LED impact on drift DONE")
run_task(led_impact_test()) |
Here is an easy way to do that: from pybricks.hubs import PrimeHub
from pybricks.pupdevices import Motor
from pybricks.parameters import Color, Port
from pybricks.tools import wait, vector
hub = PrimeHub()
hub.imu.settings(3, 2500)
# By mechanically locking these motors, this will build up heat without motion
a = Motor(Port.A)
b = Motor(Port.B)
a.dc(100)
b.dc(100)
COUNT = 3000
while True:
total = vector(0, 0, 0)
for i in range(COUNT):
wait(1)
total += hub.imu.angular_velocity(calibrated=False)
# Average angular velocity, e.g. bias:
print(total / COUNT)
Single angular velocity values are bit noisy. You'll want to average them as above to see meaningful change. For the hub I tried, there was almost no change in bias for the Z axis, but after heating up there was about 0.5 deg/s difference on the X axis. Cooling down takes longer. This is expected, but also useful. It means that things are pretty constant after an initial run in. |
Here are our options:
Unfortunately, that does not include temperature. So if we want to use temperature, we'll have to either read that separately, or perhaps use the device FIFO buffers instead of rounding as we do now. That could be worth doing, but this is a fairly extensive task. |
So 0.5 deg/s is 30 deg/min. Quite a lot and it surely can have an impact. But it is roughly what I would have expected after the LED test. If you only see this in one axis it might hint that there is quite some variation, between axes apparently but likely between hubs, too. That is good to know. I could try to measure some hubs and find a less sensitive one.
I am not in the opinion that everything must be solved within pybricks itself. This seems to be a bit of a corner case. If the data is accessible via pybricks maybe it is enough so that advanced teams can do something about it on their own.
Not necessarily; there are different procedures and the handling deviates from competition to competition even in the same "series" (e.g. WRO or FLL). Example: robots have to be deposed and turned off between "programming time" and the final competition run. Sometimes 6 teams will then perform their run on one competition table in sequence. That means that alone between the first and last team there is easily a time difference of 20 minutes. I suspect that this is enough time for the heat to dissipate almost completely. A strategy might be to find a spot somewhere at medium temperature, recalibrate and prevent recalibration from there on. It would be really nice to measure and output the temperature to validate various things ;-) I will try to look into the code and see if I can do something over the following weekend. At any rate: thank you so far! |
|
Is this DrTom/pybricks-micropython@25767bc going roughly in the right direction for getting the imu temp out of the hub? It compiles and I don't see adverse effects when running it on the spike hub. If it is correct for itself I wonder when and how often this is called because of the Preliminary results measuring uncalibrated angular_velocity (and sort of drift) on various hubs with stalled motors and LEDs fully on: quite some variation between the hubs and axes. But I would like to add the internal temp. Hence the question above. Some other question: you mentioned that the drift compensation does some sort of integration over the readouts. When are old values discarded? There is the comment
but also
I assume it completely starts over when the hub was non stationary and the reaches stationary again, correct? |
|
The So to read temperature, we'd have to also change gyro and acceleration readings, in a way that still provides consistent timing. Having temperature measurements may still not be a silver bullet by the way, since it didn't seem to affect all axes in the same consistent way. |
I was afraid of this answer. So to rephrase: where it is it will read only once. If I pull it into the for loop (not sure if even possible) it will (likely) have adverse effects on the timing and the result may be or will be off :-(
Yes, I have that im impression too. For some other reason: when heating up the system there seem to be a peak where angular velocity is off most from initial state. After that it seems to ease somewhat again (never gets anywhere near the initial values). It took quite a while to see this, I don't remember but around 1/4 to 1/2 hour (and I only saw this by accident because I forgot about the experiment was still running). While this doesn't matter for the usual use case of a few minutes it indicates that a simple look-up table with temperature and drift correction (and I was hoping for that) might not be feasible. |
|
Hi all,
|
|
The heat and temperature discussion arose because rising temperature inside the hub can cause changes in the readout and finally cause the heading to drift off. Heating the hub up by external heat source was considered (I actually made a little experiment a while ago in a convection oven, which hinted in this direction). But it was then suggested to use the hub itself to heat it up by either LEDs (@dlech ) and or the motor controls (@laurensvalk). We did that and we could observer that (the uncompensated) angular velocity indeed changes under this conditions. I would like to have a measurement of temperature, too. Firstly to confirm correlation with angular drift and put some numbers to it. Finally I hoped it could help us to fix the problem. It seems now questionable how much can be achieved by that. It makes more sense to use the existing temperature sensor inside the IMU than adding another external one. Now, the code to readout of the IMU in pybricks ist quite optimised for efficiency. Unfortunately, it seems hard to add temperature readout to it. I succeeded to read the temperature once when the IMU is initialized; but it never gets updated from there on. For continuous readout it seems that a complete overhaul of the code is required. Since it is uncertain if anything is gained by this it is unlikely to happen. This is where we stand now. |
|
I extended my initial one time temperature readout to continuous readout (it is quite a hack): It works only if remove the comments from line 396 and 397: DrTom/pybricks-micropython@5dc7237#diff-382fccca3d04c7ad3438817576499b174b524cd37426af9561f4ed4ba0c9dac5R396-R397 My intention was to force a retry about every second with lines 435-R438 |
Because it is a protothread. |
It could still be useful in principle. Even if the axes are not all the same, we could store the x/y/z bias at a few temperature levels (say, low, medium, high), and interpolate based on the current temperature. This is assuming that the bias variation is consistent and monotonic in temperature. At the moment, my focus is on Adding continuous raw temperature measurements and (potentially) restructuring the way we read angular velocity and acceleration could be a separate task to be worked on in |
|
With the temperature readout working now I can finally share some results. Scatterplots of angular velocity vs temperature for two hubs: There are significant differences between hubs and axes. Some seem to be just OK over a wide range, others seem problematic. For the most part linear regression fits nicely. Hub RG shows non linear behaviour in the beginning. That could be a side effect from my measurement procedure. I tried to increase the temperature slowly. But that didn't work quite well in the beginning here. However, this is also an indication that applying linear regression might not be simple or not be possible at all. I have not yet measured the hub we had the most problems with and also not the temperature change during an actual run. |
|
Oh, wow! I must have one of those hubs that doesn't change so much when I did my test way back when. I guess when we get the fancy calibration UI, one of the steps should be "put your hub in the oven". 😆 |
|
Very nice! Just to make sure, is this data from angular_velocity() or angular_velocity(calibrated=False)? |
|
I will publish the code and all. I need some time to collect and clean all up. It is distributed over 3 projects now. By the way: I can only do that on top of your development branch
Update: the crashes are related to the temperature readout :-( |
Keep that one. You can maybe auction it: "prime hub with very stable IMU" 😂
😂 Actually, I didn't need to do that thanks to your and @laurensvalk input. Increasing brightness of the LED lights and DC of two stalled motors gets easily to 65 deg Celsius (even higher, I stopped it at 70). It just takes some time: a test runs for about 45 minutes. Delta temperature vs delta time is the big unknown here. The diagrams do look useable because I made it quite a slow process. I am quite certain that in reality the rate is much higher. We will see. At least all of this can be used to pick some axis of some hub which works reasonably well (if this option is available). |
|
The code for the test, data and results are here: https://github.com/DrTom/spikeprime-pybricks-angular-drift-analysis |
|
Thanks again for working on this. Super helpful! There is potentially a way we could incorporate this without making it too difficult to the user. Instead of storing a single 3-axis bias persistently, we could store 2 sets of 3-axis bias + temperature and assume linearity between them. The set for the upper end may not be available at first but could be set in the background when the hub is stable after reaching a higher temperature.
Thanks for updating this - I was curious about that. If you do see issues with the new 3D integration (without temperature additions), feel free to share. Getting the I2C operations right will be a bit of a chore but might work if we properly dive into it. Your data has demonstrated that this is potentially worth it. It's been a while since we looked into the FIFO buffering, so I don't quite remember why we ultimately choose not to use it. But maybe we can use that to ensure consistently timed values and occasionally read a temperature sample in between. |
If we do use the FIFO, this does seem to allow inclusion of temperature:
So maybe it is worth double checking if round-reading also supports it. Otherwise, if we do use FIFO mode, maybe we can set this up to produce a single timestamped frame, since we don't necessarily need a FIFO buffer as such, unless this would allow us to pick a higher data rate. It looks like we would want to use continous mode with a watermark index to tell us to drain it in time. All this does look like it would give a significantly higher I2C handling overhead. |
This is from this years WRO competition. One of the teams is experiencing quite severe gyro drift. This is an advanced team with their own gyro drive function base on reading
imu.heading()and feeding the deviation with a simpleproportional controller intodrivebase.drive(). This is used throughout the run for all straight movements.The main run is composed of 4 sections, where in between those sections realignment against the boarder is performed and
imu.reset_heading(angle)as well asdrivebase.reset()is called.This works very well for the first 3 sections or when each of the 4 sections is performed in isolation. However, if all sections are run in sequence
imu.heading()drifts off significantly towards the very end of the run. We have observed deviations of more than 15 degrees. The deviation is so large that this seems not to be a simple accumulation as well as running isolated sections is inconspicuous. The deviation is always towards the same direction.So we would like to debug this (unless there is a immediate solution).
For one thing we would suggest to investigate the current state of internal drift compensation and see how it changes under
certain circumstances and influences. Is there a way to read the values (and output them to the console)? Is there a way to force the hub explicitly to recalibrate during a run? How long would we have to pause the robot so that auto recalibration occurs?
Do you have any other ideas of what we can look at?
I can rebuild the firmware and perform some simple changes like changing constants if that helps.
The text was updated successfully, but these errors were encountered: