Decoding sensor data #1
Comments
|
Actually, I had TX and RX backwards; TX on the debug header goes to RX on the ESP32. My bad (you mentioned it in the blog post and I missed it). |
|
Ha! This stuff happens, glad you got it figured out! Please feel free to ask anything else, and let me know if you end up building something on top of all this! |
|
My main goal is to build an esphome integration. I've never had to fully reverse a chip before, but it feel like a fun challenge that I might be able to help with, and I like the small form-factor and low price of the Vue. I'd appreciate any pointers if you have any; currently waiting for a USB oscilloscope to help with identifying what each of the pins are doing. |
|
Sure! First off, oscilloscopes are very helpful for identifying the function of things, but when it comes to actually reading digital data you want a logic analyzer. Anyway, please do poke at it, it's a lot of fun. If you'd like though, I briefly described the physical communication on the board here: https://flaviutamas.com/2021/reversing-emporia-vue-2#on-board-communications There's another chip on the board that communicates with the ESP32 over I2C. The raw I2C message gets dumped over MQTT as hex here: https://github.com/flaviut/emporia-vue2-reversing/blob/master/example_messages.txt#L2-L19. Note that right below that hex data are the output results:
|
|
By the way, I've given you write access to this repo! Please feel free to do whatever you wish with that access, but let's keep Emporia-copyrighted stuff off of here to avoid causing them problems :) |
|
Hey - I'm actually working on the exact same goal as @krconv. I'm mostly trying to make it so I can add a separate ESP32 running ESPHome that would read the messages on the i2c bus and publish that data to MQTT or through the native ESPHome API, without the need to flashing the onboard module. The reason being that I like having the data in their UI, but I also want to be able to do second triggers on the energy consumption. At first I was thinking about making a MQTT bridge to their AWS IOT mqtt endpoint, but I wasn't able to reverse engineer the certificates/auth method. Because of this, I've been working on reverse engineering the i2c message data and I think I mostly have it figured out at this point. Here is my data spreadsheet I used to figure out / validate the parsing logic: https://docs.google.com/spreadsheets/d/1Yu98tDRhFJqqMntNak6X297SKyzQtf0_Cvv_g7qKewk/edit?usp=sharing. The only data missing from the message is the calibration - which is only on the esp32 - and the voltage frequency. Edit: Forgot to say, thanks @flaviut for the post and the data in the repo, it was immensely useful. Also, I updated my Vue2 firmware to the last version (Vue2-1612298550, Tue Feb 2 20:42:33 UTC 2021) and it's still outputting logs and the TEST_MODE is still there and it looks like it's still working. |
|
WOW! This is incredible! I'm curious about how you went about doing this, I did spend a while looking at things in a hex editor, but didn't really get anywhere. |
|
It took me a bit to understand the spreadsheet, but now that I do, I've written up a C declaration for that data: One thing that I suspect is going on here (from my use of the Vue 2 locally) is a non-1:1 mapping of channel numbers to data indexes. I think, but am not sure, that some of the indexes here do not match up with the numbers printed on the box. |
|
I'll start off by saying that I'm mostly out of my league when it comes down to reverse engineering things, but I wanted to give it a shot anyway. Input P[V] valuesThe first thing I did was to manually check the debug/v2 message that you had in your post to see if I could find any pattern. An obvious one was the amount of FFFF value bytes separated by 2 bytes of data. I also noticed that all the P[V] values were small-ish negative ones. Doing a find and count for FFFF in Notepad++ returned 57 match. Dividing that value with the 3 P[V] per input results in 19 - the same amount of input we have in the logs. That block was then clearly the data for all the input power values. I copied the first 12 bytes (which should correspond to the first 3 P[V] of I01) in this amazing hex tool and started to check for other patterns in the data. I also copied another 12 bytes from another message with positive values, and the values seemed to correlate when parsed as INT32 (DCBA). I wasn't sure what to do then, since the decimal values weren't the same as the expected values. I decided to plot values from multiple inputs on a graph to see if there was a linear equation hidden somewhere. Here's what it gave me : As you can see, it does look like it's linear, but calculating the actual expression gave me a pretty significant offset (which can also be seen on the graph since the line is not going through 0). A small offset is expected since the values are most likely rounded, but the result is too much. Also, by looking at the graph, we can see that there are 2 general groups. I was able to see that the slope value was different between I01-I03 and I04+. After thinking about it, and checking the points (expected value and RAW decimal) on the graph, I noticed that not all the points were exactly on the trend line, so I started to wonder if the calibration numbers could be at play, since it would explain the deviation. I decided to multiply the values by their respective calibration and plot those values. Here's the graph after that : As you can see, the deviation became negligible, and the offset was almost 0 (still some but very small, > 0.03). This told me I should now be able to calculate the slope and then use that to get the P[V] value. The slope 5.5% for the first 3 inputs, and 22% for the rest. Voltage ADCAfter that, I was wondering what other values I could find, and decided to enter the next couple of bytes into the HEX tool, and got three values that I could spot easily in the debug output:
I wasn't sure what those values where, but I decided to see if multiplying this with the calibration data would actually give the voltage, and it actually did. So I figure this is a voltage ADC value. Input Current?At this point, the amount of bytes of data was pretty low, not enough for 19 x int32, I also noticed that the voltage ADC was an int16, so I started to see if I could notice any pattern with 19 x int16 in a row. It was pretty easy to spot since all the values were pretty much the same, see for yourself : I wasn't sure why there were 2 different values Well, that doesn't look anything close to 372.4... That was mostly pure luck to find that, but it looks like it is indeed how it's calculated. Just need to shift the period and we're golden! Voltage degreesI know it's pretty useless and not needed, but at this point I feel like superman, so I wanted to see If I could crack the code. The amount of bytes left is pretty limited : There's only a single block with 3 int16, so it's only logical that
The 0 degree made sense, but how about the I haven't figured out if the voltage frequency is in the data, but I don't think it is, since the first 4 bytes seems to be some kind of counter, and the last 2 bytes are always |
If you have a message that you think doesn't match the mapping, let me know. I don't believe that it will deviate from this structure but we never know. |
|
@flaviut I wanted to know - do you happen to know the i2c frequency? I'm trying to connect to it but I'm not having much luck at 50kHz or 100kHz. |
|
@Maelstrom96 I believe it's 100kHz. I uploaded a dump from sigrok pulseview to https://github.com/flaviut/emporia-vue2-reversing/blob/master/i2c%20dump.bin, and unless the time base is off in that, it shows 100kHz.
That hopefully the screenshot will also explain how it expects to be asked for data. |
|
This is awesome! I just had fun decoding the message too, and it's cool to see that I landed in the same place. I used Python to help me walk through the message, activating each of the inputs and finding the bytes that changed. In case your curious, this is my helper script. I was just about do try to figure out the calibration as well, and ended up with a linear relationship (just eye-balled it, didn't verify it was exactly linear) between points on I04[V2], similar to you. From my testing, I'm pretty sure the uint16 with offset 0x00EE is the frequency; it was sandwiched between the AC voltages and degrees, and it was consistently changing with the frequency output in MQTT. The conversion for my device is For calibration data; I think we could do the same thing that the existing CT clamp sensor in ESPhome does, where whoever is flashing the device would need to manually calibrate each of the sensors. Maybe we could reverse engineer the calibration data from nvs, and make that easier? I think it's worth skipping that for now. @Maelstrom96 Would you want to work together on the integration? I'm not familiar with building an ESPHome integration, are you? I am familiar with C++ and think I could make faster progress on the integration though if you want to keep looking into the connections on the board because I'm not as familiar with the signal processing side of things. |
|
And I'm pretty excited...I feel like this will be a huge improvement for cost effective energy monitoring if we can get it to work |
|
Reopening for visibility, to make it easier to find for others interested |
|
@krconv I'm not really familiar with ESPHome components. I tried to see how ESPHome handles I2C devices and I'm not sure if we're going to have an issue with the TwoWire lib that they're using. The reason being that is looks like requests read buffer is limited to 32 bytes with TwoWire, and we need to query the full 284 bytes all at once. |
|
I've been trying to learn how various ESPHome components work. I think under the hoods, i2c will use the TwoWire library by default, but we can configure it to talk to the onboard i2c controller instead and I'm guessing that's how the the original software does it. I'm not sure what other implications this has, but this is an example from here to tell ESPHome to use the native ESP API: esp32:
board: esp32-c3-devkitm-1
framework:
type: esp-idf
version: recommended |
This is needed since in some applications, we need much much larger buffers. See emporia-vue-local/emporia-vue2-reversing#1 (comment)
|
We'll most likely have an issue with https://github.com/espressif/arduino-esp32/blob/master/libraries/Wire/src/Wire.h too. And if you look at |
|
I tried out that approach, and agreed I don't think it's much of an option. I think setting the framework type to From what I understand, Arduino is a generic framework, and arduino-esp32 are the bindings written to port Arduino to ESP32. ESPHome just barely added support for using the native esp-idf framework in release 2021.10. It seems like if we could get @flaviut's PR merged to the esp8266 and esp32 Arduino libraries, we could keep doing I2C through Arduino, or if we have enough support on esp-idf, we could go in that direction (because this component would only need to be used on esp32 anyways). Also, I've been thinking about a potential Config format; have either of you tried that out yet? Any thoughts on this idea? Config Ideasensor:
- platform: emporia_vue
update_interval: 2s
phases:
- id: phase_a
phase_input: BLACK # i.e. the color of the wire that is connected to the phase
- id: phase_b
phase_input: RED
power:
- id: total_phase_a
phase_id: phase_a # not sure how the factory software figures out which phase to use; this makes it explicit
ct_input: A # uses A-C, 1-16 as labeled on the outside of the Vue
filters: # and each one would have a separate calibration
- calibrate_linear:
- 0 -> 0
- 2393 -> 2.0
- id: kitchen_outlets_power
phase_id: phase_a
ct_input: "1"
filters:
- calibrate_linear:
- 0 -> 0
- 2393 -> 2.0
- id: stove_power
name: Stove Power
phase_id: phase_b
ct_input: "2"
filters:
- calibrate_linear:
- 0 -> 0
- 3438 -> 2.0
- lambda: return x * 2;
I've been testing out config validation on my fork using this external component: external_components:
- source: github://krconv/esphome@add-emporia-vue
components: [ emporia_vue ]
refresh: 1minI haven't tested anything on hardware yet, but my next step might be to connect a spare ESP32's i2c port to the Vue and see if it can talk to the sensor at all with the esp-idf library |
|
The real calibration constants are However, the values here are surprising. The interesting data is marked as erased, while the uninteresting version is not: Decoding these gets you basically what you expect: 19 entries, the first 3 of which are different than the rest. I bet that these can be decoded as: |
|
@Maelstrom96 You're right--increasing the number there won't work unless we adjust the function signatures too. We already have https://github.com/espressif/arduino-esp32/blob/master/libraries/Wire/src/Wire.h#L98, and it's a real shame that that function is written in terms of the other functions rather than having all the other functions call it. However, it seems like a simple-enough refactoring, and the ESP32-arduino project seems responsive to contributions.
:( I guess you could try adding MQTT support to the esp-idf version of the esphome project, that'd probably be something useful for everyone. But I think the fastest & easiest way would be to do PRs to the wire library, since they're pretty simple. I can do that over the next day or two!
Huh. Neat! I had no idea ESPHome made things that easy, that's really nice! |
|
The calibration data I think will be helpful, the more I think about it. Calibrating 19 CT clamps manually might be a real pain; maybe we could provide a script that helps parse the calibration data later on. That sounds great; I agree getting MQTT on esp-idf would probably be helpful for everyone, but ya it might slow things down. I also checked the api component, and that relies on Arduino too; so there would be no way to get the sensor data to home assistant without some more development. I did find a native MQTT library; if we do go that route, we might be the first integration that requires esp-idf, which might have some other implications. I'll take a look at the work involved to support MQTT on esp-idf and see if the i2c using the native library even works for this scenario. |
|
Ok that makes sense. I haven’t purchased a usb to serial adapter yet. I wanted to make sure I could understand it all before spending money to dive into it. |
|
Exciting little project... I received my Vue last week and followed flaviut's "Reverse-engineering" guide to get it talking to mosquitto / node-red running on a Raspberry Pi via MQTT. The parse_mqtt_dbg.py is pretty straight forward and outlines how to process the MQTT messages into an InfluxDB, which I am looking to reimplement in node-red. I notice that it's over 4 months old. Is there any newer code / documentation available which outlines the processing of the MQTT messages? Or is that pretty much all there is to it??? The circuit breaker on my hot tub trips a couple times each winter and I almost lost the hot tub last year due to the breaker tripping (it was well below freezing and the power was off for a few days). By monitoring the power usage I can get an alert as soon as the breaker trips again... Thanks for your hard work and dedication to this project. It really opens up this sweet little device as a powerful home automation tool. |
That's all there is to it. The reason there hasn't been any more work on that is that running fully open-source firmware is less janky (doesn't require specific IP addresses, working with copywrited firmware, or any limits on the wifi password). So all effort has gone into the FOSS firmware, through ESPHome.
Wow! That's an awesome use, and I'm glad your hot tub survived! Maybe you could use another esp32, duct tape, and one of these to make yourself a self resetting breaker 😁 |
|
lol... I'll see what I can do about the automated breaker resetter... Thanks again... You guys rock! |
|
I've finally taken the plunge and ordered mine yesterday after finding this thread. Just wanted to say thanks for all the hard work, brilliant guide and give a quick bit of feedback on how things went for me.
I've hooked up one of the big CT clamps (that's all we need here in the UK for a normal domestic supply) and one of the little CT clamps on channel one. Readings come through ok and look correct, so far everything seems stable and I'm going to watch it for a bit before I take a deep breath over the weekend and try to squeeze all those 16 CT clamps into my consumer unit. |
|
It looks me a while to get mine installed, mostly because I was daunted by the task of opening the load center. Once I started, everything went fine. Thank you for the work you guys have done. I directly uploaded the yaml with @krconv's branch, and it's working smoothly. One thing I noticed which might help someone else to troubleshoot: If your power values see-saw between positive and negative, you have mapped the circuit to the wrong phase. I've attached a screenshot of my graph, showing before and after I fixed it to illustrate. @flaviut's instructions are very clear on how to avoid this in the first place. My excuse is that I have a cluttered load center and somewhere along the way, I started counting wrong :) |
|
Glad you got things working :) I want to point out one thing: it's not unusual to get ±1W of noise in the readings, that's normal and expected. The problem your image shows is -0.25 watts (on average). Unless you have a solar system or something, that's a problem:
|
|
Just stopped by to say I have converted all four of mine and they are running stable. These CT's can be finicky and I recommend calibrating them properly. The default settings were not accurate for me and depending how how close the CT is to other wires and the panel, I see a lot of variances that need to be filtered out. I chose a median filter, sliding window of 5, with a report rate of 1. I also added a lambda filter as it bounces below 0 quite a bit (a sign that you don't have it 100% calibrated) and I just want to filter those out. The funny part is I am using emulated kasa to feed some of these circuits to my sense, just because I like their UI better. |
About how wrong was the data for you? I'd expect up to about 5% difference because we don't set a certain calibration constant dynamically at this time. I'm curious if this was the problem.
Do you find this works better than a sliding average like in the example? I've ended up with this in my personal config, but if there's something better, I'd be happy to change: and
I disagree. Every system has some degree of noise to it, nothing that exists is perfect. On a circuit that consumes 0W, I'd expect to see 50% of the readings below 0W, and 50% above. Are you seeing more than ±1W of noise around 0W? For reference, here's what I see with the filter above, and I think it's reasonable:
That's really cool. That's the kind of stuff I love about hacking home automation stuff. You don't need to deal with every vendor's walled garden :) |
I am seeing about 5% variance now, but originally all of my numbers were trending low when compared to other manufacturer devices monitoring the same line. When I converted these units they had been running for a while, and the calibration numbers I was seeing in the logs/NVS were higher than the defaults. I believe in both cases I was seeing different values from defaults in the source ( 0.023109 for phase A and a different number for phase B that I forgot to write down). I am assuming this was fine tuned by emporia.. I also have other CT's monitoring these same lines (The Sense, CT's in my powerwalls, my EV wall controller is also capturing them). I slowly adjusted the values until the vue started reporting similar numbers for voltage and wattage.
I think it comes out to a wash in the end. I am running with your filters right now (changed the windows to still report once per second) and I don't know if I would be able to tell the difference. /shrug
In some of my lines I see it jump ±20W when at 0. When I first installed my sense, their support contacted me and let me know that if the CT was pressed against the side of my panel (rigid wiring) I would see behavior like this. I am assuming my issue is similar there for these. I am going to have to open up my panels and see if I can find a more convenient spot. The stock settings gave me low numbers originally. I think it was showing my voltage around 10V lower than it actually was, which I think was driving all of the power numbers down. Once I had that calibrated, I am seeing something similar to you in that a 0W reading bounces around ±1W.
I am loving this as well! Once we have arduino framework support, I am looking forward to having the device perform the kasa emulation itself. :) |
|
One more data point about the inaccuracy of the readings - one of my circuits was showing abnormal fluctuations (+/- 30W with no load) after I bottled everything up. I went back and opened the panel and I saw that the 2.5mm jack had slightly dislodged from the port because of all the wires that are crammed in there. I reseated the connection, and everything went back to normal. Obviously, this isn't a code issue, but for someone troubleshooting inaccurate readings, this is another troubleshooting step to consider. |
That's right, I remember now! I had this exact same issue, and actually put off putting the panel cover back on (since it would move the stuff inside around) because of it. Thanks for bringing this up, I'll add it right to the FAQ. |
|
Thanks for all the hard work here. I flashed stock Micropython to the esp32 and can read the CTs over i2c. Here in the UK, typically we have only a single phase. I was wondering if I can read the power factor of the circuits, and report kWh as well as kVA(h). |
As far as I'm aware, no. There may be some configuration options for the other MCU on the board, but I've never seen them so I don't know if they exist. |
|
Is anyone having any long term stability issues? I am using Looking at the logs, I see this a lot: I've tried rebooting a few times and have already reflashed the firmware. I don't see how running esphome can cause this since we are only reading data from the bus, but I am curious to see if anyone else has also seen this error. |
|
I see that same error some of the time and it usually doesn't go away without actually disconnecting the power for about 5 seconds and plugging it back in. Since mine is two phase, both circuit breakers that are providing power need to be off at the same time. Also, you might have more luck with |
|
I've generally been supporting this here, if you'd like to move the conversation over. @nightweyr I'm really not sure what could be going on here--the error is
I'd try a few things, one at a time and then maybe together:
|
|
Thanks for the suggestions! I was putting off manually powercycling the device since I'll have to open up the panel. I'll try it this evening and post my findings in the comments of the gist page. I was able to remotely modify my esphome to use @flaviut's fork, but that didn't help. |
|
Hi! There are 3 power values for each clamp but only one seems to be useful. Do you know why all 3 are provided and what they could represent? I mean, I'm really curious to understand how the Atmel IC is computing them. For instance, the power value for the opposite phase is often close to the negative value of the power, but not always. Somewhere, either in the ESP32 firmware or the cloud, Emporia automatically figure out which of the 3 value they should use, but why aren't they doing this directly in the Atmel firmware to avoid sending useless data? It's not super useful to understand all this as you have a working solution, but for some reason, I can't bear not fully understand how something works! |
Power is current * voltage. Each of the power values is for a different voltage. Keep in mind that AC is system where current and voltage continuously change--it's not enough to simply multiply, you need to multiply over a sufficiently small time interval.
You are likely on a north american split-phase system. Your phases are opposite, ignoring tolerances, resistance losses, etc.
Same reason the ESPHome plugin doesn't do this, and you need to set this up in the config. You need historical data, because the direction of flow can change, for example, in solar systems. It's also hard to do for circuits that stay around zero load. |
Yes, I know and that's why I found the values in the example_messages.txt confusing. If the two phases are exactly in sync but with opposite voltage (as they should be), the power value for both phases should be the same, but one value positive and one negative (assuming that the sampling rate of the ADC is high enough). That's not what I was seeing in your document. For example, the 3rd and 13th row are: Anyway, I tried to dump the raw power values in my setup and they are exactly as I would expect them. The power value for phase A is always very close to the negative value of phase B for all clamps and that makes sense. I'm not sure why you got weird numbers in the example_messages_txt file, maybe you have a different setup (but the voltage and frequency points to a North American setup too).
That makes sense, thanks! |
I don't know if that's the case for me. I live in a condo, so the whole electrical thing gets a lot more complicated. I might just have 2 phases out of 3. Actually, now that I look at it, I'm sure I have two out of three phases. My V2 is offset from V1 by 120 degrees according to that same file. It'd be 180 for split phase. |
That is very strange, I didn't know it was possible! Split-phase systems are supposed to be 2 lines that are out of phase by 180 degrees (that's what the wikipedia article you sent explains and that's what I've always seen). If you really have 2 out of 3 phases with an offset of 120 degrees, that could be a problem. It means that all your 240v equipment is working at only 75% of their rated power! Do you have a well defined load on one circuit, like a 1000w heater or something like that? You could look at its power usage reported by the Vue and see if it really is 1000w or closer to 750w. |
Yup. I guess you're right. I never thought about that. Regardless, all the appliances are rated to work at both 240V & 208V. That's what I see when I check random manuals for various appliances. |
|
Yeah, they will work but may have lower performance (oven/water heater will take longer to heat for instance). I found a thread explaining that this is not common, but it can sometimes be seen in condos or in some areas. That's interesting, I didn't know about this! |
|
In the US it is single phase or three phase. Most people believe there are 2 phases in a single phase scenario but there isn’t. It’s just two separate transformer windings that are in phase with each other. There are some old plants that have two phase 5 wire systems but it is very odd and I only have ever seen it once. But a residence in the US is most certainly single phase unless it is a farm or something.
On Tuesday, April 5, 2022, 10:07:39 AM CDT, Francis Pagé ***@***.***> wrote:
Yeah, they will work but may have lower performance (oven/water heater will take longer to heat for instance). I found a thread explaining that this is not common, but it can sometimes be seen in condos or in some areas. That's interesting, I didn't know about this!
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
Hello
|
|
Hey @BlagovestKambarev, let's discuss this in #8. For anyone else, locking since this particular issue is resolved, but still very interesting, so I'd rather not close it. If you think I should unlock it or you have something else to discuss, please open a new issue! |
Repurposing this issue for the discussion that it started. Was originally a question about connecting, and turned into a discussion about deciding the sensor data.
In the blog post, you mentioned that one of your USB-Serial adaptors wasn't working when you tried to connect to the ESP32. I'm wondering if I'm hitting the same issue; I only have one serial adaptor to test with at the moment though.
I'm using the FT232RL FTDI Mini USB to TTL Serial Converter, and getting nothing while trying to connect:
Process I'm taking to enter flashing mode:
But that's not working; my only thought is that maybe there is something else connected with TX/RX that is interfering? And a different serial converter could sort through the noise?
Any guidance would be appreciated!
Edit by @flaviut for visibility:
There is now a ESPHome component. You can follow development at
The text was updated successfully, but these errors were encountered: