-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Moving average for Power Measurement #4727
Comments
Hi, Your idea of doing some type of data processing for the power measurement is nice, but there are some points to be taken into account: If the 15W bulb you are measuring is a cheap one, it will have a lot of noise that it is being injected in your mains AC. Also if you have other devices of that type, you will be having much more noise. There are some of these cheap ones that have a noise switching power supply and I saw others that have a capacitive power supply that produces a huge amount of noise that it is just an additive noise, not a random noise. So, the moving average filter won't work in an additive noise environment. If the noise is just white noise type or pink noise type, a moving average filter will do the job. In an environment where you have an asymmetrical noise, a moving average filter will give you all the time a higher value than the real one. An IIR or FIR filter will do the job but only if it is tuned to be a low pass filter to filter those noise spikes. Anyway, the sampling rate for power measurement needs to be adjusted to be a multiple of the switching frequency (this needs to be adjusted experimentally). So, as you have explained that this power fluctuations only happens in those devices but not in for example a Sonoff Pow, and that this needs a little more complex processing rather than just a moving average in order to have a little more accurate measurement, I think it is not going to be added soon. I would like to know other opinions also. Anyway, your idea and your request is nice. A workaround for now so as to have more precise power measurements is to use devices like sonoff pow R2 or the PZEM sensor supported by Tasmota. Thanks for sharing your ideas 👍 |
Hi, |
I notice the same behaviour as you do. Most energy sensors are kind of stable but my shp6 acts the same as yours. |
OK so what would you suggest ? Do you think implementing some kind of prefiltering in tasmota is an option or do I simply need to live with the poor measurement result of the SHP6 ? |
As a low power measurement takes some seconds you would need to calculate the average over 12 seconds. In that time the load could have been switched off or changed so that makes it very dificult to calculate. I'm afraid we have to live with it unless some clver dicky finds a quick filter option. |
Yes of course that's the price for it. Nobody will find a quicker filter. I already wrote it before : "Obviously the refresh rate of the data shown in the webinterface will be lower, but getting an update e.g. every 10 or 20 seconds would be acceptable for me" So I currently see 3 options
|
I do this with my own code on my server. There's no need for a delay to collect measurements; just use a rolling average. set lastMeasurement to saved value, or zero if none. This will cause a ramp-up for the measurement over the measurement period, and a ramp-down when usage returns to zero. Fluctuations are smoothed over the samples. I use this with temperature and humidity readings as well, since they can similarly fluctuate. An example in bash:
Edit: decreased the number of samples to speed up the convergence, and added an example of the ramp-up/down. |
Your method is really good for smoothing data with white noise. If you have unbalanced noise, that method will show a value greater than the reality. For example if you have a 15w bulb, so 15 should be the real value, but if that bulb is noisy with spikes only greater than 15w like 15, 18, 15, 15, 16, 14, 19, 16, 15, 15 Your smooth value will be higher than the real 15 because of this spikes in the measurement. If you have a IIR tuned as low pass filter, you should have a more accurate value for that case. Anyway, your algorithm will show a more stable value than the actual raw measurement. |
to eliminate spikes in a test series do not use average calculation, use the median: 14, 15, 15, 15, 15, 15, 16, 16, 18, 19 (=15) and everything is fine. Works also with raising/falling value. Disadvantage of all statistical methods is the time delay of between real value and displayed one. |
Correct. And (at the moment) in order to apply any statistical method (smoothing), all the data needs to be transfered from the front end (socket with tasmota) to the server which performs the filtering. Instead of transfering raw data through e.g. MQTT to do the filtering on a separate server, this should be done right at the frontend. |
A resistive load such as a 15W light bulb may also not remain 15W due to fluctuations in temperature and of course the source voltage. Also, calibration with a 15W load on a device that can measure 2.2KW may not give an accurate enough value for calibration purposes. 15W of 2200W = ~0.6% so it would be nearly impossible to get an accurate calibration figure from this. |
@andrethomas : Calibration at a single load and the resulting precision within different load ranges is a completely different issue. |
As @andrethomas said, this is a hardware limitation, so any data processing added to this will make a marginal improvement to the measurement and also to be used only for lower power measurement values. At higher power values, this noise should not be present. Anyway, the median approach from @curzon01, I think that will result in a more stable and real measurement and also faster than IIR or moving average methods. I don't know if that processing will be better to be added in Tasmota or in the home automation software itself. |
Technically incorrect - they are related. I do not think implementing filtering on Tasmota side is viable as filtering strategies will change depending on use case - i.e. resistive load vs. inductive loads, just to give a simple differential example... lets not forget about switch mode loads :) So my recommendation is not to implement filtering on Tasmota side at all. |
From the technical side and calibration measurement theory, a one point calibration should be done close to the maximum rating in order to have the minimal error all over the range. So, if you calibrate any device with a low value, the error is much bigger than calibrating from a bigger value. Anyway, the spikes in the raw values is indeed as you explained a hardware limitation. |
In very inert systems (temperature for example) it is not a problem to do first transmit the data to the home automation system and then do the filtering. Power measurement is different. Of course loads can change quickly, however, I think that a time resultion of 10-20 seconds is an acceptable tradeoff to get rid of these nasty fluctuations. |
Your opinion. I reject this from a technical perspective though :) |
Ideally, for precision , we should calibrate at a few points e.g. 0.1W , 1W, 10W, 100W,1000W. |
I don't want a Wifi oscilloscope.. |
Let me know when you find one that does not require hiding its inefficiencies by averaging output data :) |
I fear we won't see such a device at a reasonable price in the near future. |
I think there's an important part which you are not completely grasping though. Below is my mains input as measured by an industrial grade voltage sensor. Over a 24 hour period an hysteria of up to 5% is observed on the AC supply voltage to the house. Lets assume a 50W light bulb for calculation purposes, which is rated at 50W 220V so theoretically should draw around 227mA @ 220V meaning an effective reactive resistance of 968 ohms. If we apply ohms law and bump the input voltage by 5% up to 232V and assume the internal resistance of the bulb remains 968 ohms this would translate to ~240mA - or recalculated to a wattage would mean ~55W - Aside from the additional unwanted 5W of power what is more interesting to note is the physics behind the math and what the math tells us about the physics - we bumped our voltage up by 5% but our power demand increased by 10%! Weird, huh? In reality things are a little different though. If you ever find yourself in posession of a variable transformer you should take the oppertunity to play with it a little and see what kind of hysteria is injected in your theoretical model of how things should work in an ideal environment. If you were not a programmer you would get terribly confused by the following if statements (which actually mean the same thing)
So I'll leave you with that and, of course, that adding a software filter to fix a hardware problem has no place in the firmware itself... you may of course post process the data in your HA software by all means necessary for any particular use case. |
I don't think so.
This is only weired for people living in a linear world.
Sorry I don't understand what you want to tell us with all the rest so I won't comment on it.
Intelligent sensors do exactly perform that task : Data prepreprocessing which in many cases means data smoothing. But, useless to discuss this further more, we obviously have different opinions about that.
I explained above, why this is not a good solution: Massive raw data transfer to the home automation server. I don't understand why you're fighting so enthusiastically against my suggestion: Filtering should be an option, not default. If it's implemented in the software, you are not obliged to use it. |
Unless you're sending tens of thousands of MQTT messages per second you won't even make a measurable impact on a Raspberry Pi A+ running mosquito and node-red.
I'm not fighting it, just questioning the need for it. Make a PR if you feel that way inclined :) |
This post is interesting. Let's summary ideas so far:
The devices Gosund 111 have a really bad repeteability of power measurements for lower power loads < 100w. The rest of power measurement capable devices and sensors supported by Tasmota have a better resolution.
In HA software: Can be added any type of processing and representation customized for each use-case. IIR, FIR, moving average filters, statistical median or others. (ACTUAL WORKAROUND) In Tasmota: Can be added a simple processing that uses few flash and few RAM and don't block Tasmota for other tasks.
In the case of adding to Tasmota seems that a statistical median would smooth the readings, limiting spikes and having a close-to-real values. Could be taking 9 samples (
Theo is the owner and the one deciding to add it or not. So, I think he was clear saying that it is needed that someone can prove a fast and light method to improve those devices. So, this request is on hold up to someone would like to program, test and prove a enhancement for those devices. As a workaround for now is to use another power measurement device or a PZEM sensor for more stable power measurements lower than 100w, or process the data in your home automation software. Thanks everyone for sharing theirs ideas. Very interesting and very appreciated. |
Thank you Adrian for your summary. So if Theo as the owner decides to add, we still need a volunteer to implement such a routine. I think that the small simple median-like subroutine as Adrian mentioned above is already sufficient. Or any other filter, algorithms are standard and C-coded examples can be found on the www, e.g. https://atwillys.de/content/cc/digital-filters-in-c/ However, I won't be the volunteer to program that. I can't. |
@Jason2866 YOU'RE RIGHT!!! just did calibration with 75W + 75W + 25W. Then if one or two is turned off it tries to show the correct value. It's not that close as the SHP2 but it's near enough. I stop further analyzing regarding non-linear/lineair and call it a day concerning SHP6. |
Yes I already mentioned that much earlier. What do you think about using a 10mOhm Shunt instead of a 1mOhm ? |
SUCCESS ! with a modified SHP6 I changed the 1mOhm SMD shunt against one with 10mOhm / 2 Watt (WW25NR010FTL) Result => Big question, is that still safe ? =>
So even the most cautious user can use such a modified SHP6 up to 727W which is sufficient Maybe some experienced user can measure internal temperatures (DS18b20 ??) with the old shunt |
Nice hack. Perhaps one could try using a 2mOhm shunt for less dissipation and perhaps more stability too... |
I wonder if the one that ships with the product have a 1% tolerance... |
the shunt tolerance does not contribute to fluctuations at all |
I now tried to measure without the new averaging/mean function, i.e. I flashed an old version of Tasmota (6.3.0.5). The old tasmota also returns stable results. |
Great. So you ruled it out that is 100% the hardware |
I just flashed a SHP6 with 6.4.1.12. With no load connected it reports: |
Did you calibrate the device ? |
@SirJMD |
I'm about to calibrate it. Just wanted to post some out-of-the-box numbers. Rather crazy numbers with no load. |
Hello, I've been reading this thread, and although I'm not nearly qualified for this type of "language", I've a very vague question that I'd like to see it clarified: to achieve correct measurements from this device it's necessary any hardware change (besides the operation of flashing the tasmota firmware) or it's only a software operation? |
@flrsilva Reported above means changing the surface mount shunt to a higher resistance value yielded more stable results with this particular device. |
@flrsilva : The current tasmota version includes a data smoothing algorithm and if you use a Blitzwolf SHP2 or Gosund SP1, the precision of power measurements with Tasmota and no further hardware changes is good enough ( for Joe Doe users). If you intend to use a Blitzwolf SHP6 or Gosund SP111 it's highly recommended to change the 1mOhm measurement shunt with a bigger one. This SMD needs to be changed |
With no hardware mod I can't get the SHP6 to produce stable measurements. I can get it calibrated to be stable and true at high or low load, but not both. Calibrated to 1950W ohmic load was straight forward, but at no load it would report 100-150mA purely reactive. Calibrated with 40W ohmic load would result in high loads being way off and having 10-20% reactive power component. |
I can confirm that. This is already described above. But if you use a 10mOhm Shunt you can calibrate at e.g. 60W and you get fine results for both low and high loads (e.g. 5W and 1kW ) |
The problem with 10 mOhm is that at 10A you dissipate 10 times more than with 1 mOhm, That's quite a alot extra, especially when they already try to mitigate heat from the ESP8266 with a thermal pad. I have several values of shunt resistors on the way in the mail, and three additional SHP6, so I'm gonna test a few different values and see what works best. |
I've been following this thread for a while now. And I'm seeing some really interesting progress here, really nice to see how much effort people put into this issue! But I was wondering: I'm aware that best results are given to test the diffrent SHP's with the same load ofcoarse.
(there can be a slight difference in Voltage, but just took 230 for the ease of use. so the power calculations can be a bit off) |
So, I was bothered by the bad measurements aswell and took the route of modding the shunt. Put in a 10mOhm resistor and I get very good results when calibrating with a 60W (which measures 55W with a clamp current-meter). I hooked the modded plug to my portafilter coffee machine which pulls a solid 1.8kW/8A when heating. No issues with the temperature of the device so far. BUT, when calibrated to 55W I only get a measurement of around 900W. Other way around, if I calibrate to 1800W, the 55W bulb will show around 100W. Can you please try this yourself and confirm I am not doing something wrong? The commands I use for calibration are Voltageset, Currentset and Powerset |
Hello, |
Here are the values reported by my SHP6 plug during a 3d printing session. I haven't made any HW mod to the SHP6 plug. I only flashed Tasmota v6.5.0. Can you please advise if replacing the 1 mOhm SMD resistor with a 10 mOhm one, would resolve the power fluctuation issues? Thanks, |
Once heated up the hot end and the bed (if you use a heated bed) will need to remain on whatever temperature setting and these are usually controlled by either intermittent on/off signals or more commonly by PWM - The latter in conjunction with the likeliness that you are using a switch-mode power supply and the fact that the power draw from the stepper motors would not remain constant throughput a print I would say that the results you got are justified.
Reducing the shunt resistor will improve accuracy especially on lower current values as it allows for better calibration and measurement results but this will not exempt the device from the laws of physics it is being subjected to as outlined above. |
@andrethomas - thanks for replying. I'll do a test with the heated bed turned off. I'll also do another test with a Xiaomi smart plug. Has anyone tested the SHP6 plug with the original factory firmware, to see if the power variations appear as well? |
I think AndreThomas could be perfectly right, I get similar plots for some devices. I used the functions powerdelta and powerlow in order to see only "relevant changes"
This is explained here : #4905 Ralf |
Meanwhile I did another test: I have turned off the bed heating during the printing process and here's how the power consumption looks like: The spikes are caused by the heating bed circuit and there's nothing wrong with the power reported by the SHP6 smart plug. Everything is clear now, thanks again for helping out. |
I recently moved from ESPurna to Tasmota and discovered a similar problem with these devices. When the switch on a load intermittently a of the chart measurement takes place. The problem was not there in ESPurna. I think a good solution to get rid of this behavior is waiting with measurements a short period after switching. This doesn't mess up the measurements and only filters out wrong "inrush" peaks. For example my fridge on inrush is being measured >25A and peak normal power is never higher than 90watt so that peak is for sure wrong. There must be a variable where the switch_on time is stored in then it should be simple to change the sensor code to not sense x microseconds after time in switch_on |
The on peack of a motor is in deed infinit, but just a few miliseconds.
That was the reason they invented the inverter AAC, theem slow diwn and
never start/stop to avoid start inrush.
El El jue, 11 de ago. de 2022 a la(s) 05:59, Albert Weterings <
***@***.***> escribió:
… I recently moved from ESPurna to Tasmota and discovered a similar problem
with these devices. When the switch on a load intermittently a of the chart
measurement takes place. The problem was not there in ESPurna. I think a
good solution to get rid of this behavior is waiting with measurements a
short period after switching. This doesn't mess up the measurements and
only filters out wrong "inrush" peaks. For example my fridge on inrush is
being measured >25A and peak normal power is never higher than 90watt so
that peak is for sure wrong.
—
Reply to this email directly, view it on GitHub
<#4727 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXBW4K7ULDAIGJ53SRFOH3VYS6FJANCNFSM4GMCLY5A>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
@rasti70 : I realize this topic is quite old by now. Since I have a similar scenario than the one you described, have you found a solution that addresses the problem? |
Have you look for this feature in other issues and in the wiki?
=> YES
Is your feature request related to a problem? Please describe.
=> I have a few power measurement devices (Blitzwolf/Sonoff/Gosund) and noticed that some of them show really big fluctuations, e.g. I measured a 15W bulb and get measurements values of 13-19 Watts. This happens in particular with Gosund 111 / Blitzwolf SHP6 . On the contrary, Sonoff Pow, Gosund1, Gosund SHP2 seem to be pretty stable (fluctuations of +/- 1 Watt only)
Describe the solution you'd like
A clear and concise description of what you want to happen.
=> I would like to have a moving average filter. The current sample rate or update rate of the values shown in the webinterface seems to be around 1-2 Hz. I would suggest a moving average filter which calculates the average of e.g. 10 measurements before displaying or publishing it. Obviously the refresh rate of the data shown in the webinterface will be lower, but getting an update e.g. every 10 or 20 seconds would be acceptable for me. This filter could be adjustable through the console, e.g. "movingaverage 0" or "movingaverage 50" . Maybe an IIR filter also will do the job.
Describe alternatives you've considered
=> I tried to implement a moving average filter in home automation software FHEM.
This works, however I had to set the teleperiod to a very low value (10s) which imposes
a lot of traffic on my mqtt/fhem server
Additional context
=> I posted this on fhem forum, too :
https://forum.fhem.de/index.php/topic,90220.msg875650.html#msg875650
In the subsequent posts and elsewhere on the same forum I found at least 3 users more which had the same problem I had.
(Please, remember to close the issue when the problem has been addressed)
The text was updated successfully, but these errors were encountered: