Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does not work on HA Core 2022.05. #129

Closed
Murianwind opened this issue Apr 27, 2022 · 54 comments · Fixed by #133
Closed

Does not work on HA Core 2022.05. #129

Murianwind opened this issue Apr 27, 2022 · 54 comments · Fixed by #133
Labels
bug Something isn't working.

Comments

@Murianwind
Copy link

Environment

  • Home Assistant Core release with the issue: core-2022.5.0b0
  • This custom component release with the issue: 2.2.1
  • Last working this custom component release (if known): core-2022.4
  • Operating environment (Home Assistant/Supervisor/Docker/venv): Home Assistant Supervisor 2022.04.3 / Home Assistant OS 8.0.rc3

Describe the bug

The component does not work on HA Core 2022.05.

Configuration.yaml

Add your configs here if any.

- platform: average
  name: "winter"
  entities: sensor.external_heatindex
  duration:
    days: 7
  end: "{{ now() }}"
  precision: 0

**Steps to Reproduce**
<!--
  Please list the steps needed to reproduce the issue
-->


**Expected behavior**
<!--
  Insert a brief description of the expected behavior is
-->


**Debug log**
<!--
  To enable debug logs check "Troubleshooting" section of README of this component
  or this https://www.home-assistant.io/components/logger/
-->


Add your logs here.

Logger: homeassistant.helpers.entity
Source: custom_components/average/sensor.py:491
Integration: average (documentation, issues)
First occurred: 04:23:48 (28 occurrences)
Last logged: 04:37:14

Update for sensor.winter fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 518, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 754, in async_device_update
    await task
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 192, in wrapper
    result = method(*args, **kwargs)
  File "/config/custom_components/average/sensor.py", line 305, in update
    self._update_state()
  File "/config/custom_components/average/sensor.py", line 491, in _update_state
    item = history.get_state(self.hass, start, entity_id)
AttributeError: module 'homeassistant.components.recorder.history' has no attribute 'get_state'

**Additional context**
<!--
  Add any other context about the problem here.
-->
@Murianwind Murianwind added the bug Something isn't working. label Apr 27, 2022
@DavidFW1960
Copy link

Same here. Totally blown up

@DavidFW1960
Copy link

image

@Murianwind
Copy link
Author

Logger: homeassistant.helpers.frame
Source: helpers/frame.py:103
First occurred: 15:11:40 (1 occurrences)
Last logged: 15:11:40

Detected integration that accesses the database without the database executor; Use homeassistant.components.recorder.get_instance(hass).async_add_executor_job() for faster database operations. Please report issue to the custom component author for average using this method at custom_components/average/sensor.py, line 477: history_list = history.state_changes_during_period(

@AdmiralStipe
Copy link

Seems to me that developer stopped maintaining the repo - he didn't respond to my previous issue regarding database executor also. So I'm deleting the addon :(.

@slimpdev
Copy link

slimpdev commented May 4, 2022

any alternatives?

@AdmiralStipe
Copy link

any alternatives?

Not that I know of... but as I already had to change a bunch of automations and templates in HA as average sensors stopped working (I had a good dozen of them) I'll now simply remain this way. Averages were useful, but not worth the hassle regarding latest issues.

@DavidFW1960
Copy link

any alternatives?

Not that I know of
I use this a lot as well. I switched to the statistics integration but it's a very poor alternative,

@SupremeSports
Copy link

That's very sad as this integration allowed to smooth out values or averaging multiple sensors. Need a replacement solution for this :'(

@Shaun-Harrison
Copy link

Need a replacement for this too!

@bruvv
Copy link

bruvv commented May 5, 2022

I am in no means a python expert but looking at the line where it is failing: https://github.com/Limych/ha-average/blob/dev/custom_components/average/sensor.py#L491 I found out that that function get_state is deprecated and removed: home-assistant/core#70830 (and: https://community.home-assistant.io/t/history-stats-this-is-a-deprecated-function-get-state-and-state-changes/336484)
Not sure who has the python skills to fix this but that will be amazing if you could! @Limych just tagging you in hope you are getting a notification :)

@NapalmCandy
Copy link

That's very sad as this integration allowed to smooth out values or averaging multiple sensors. Need a replacement solution for this :'(

For averaging multiple sensors you can use min_max integration.

For an average you can use statistics->mean but I already have problems with large averages (+24h) since 2022.4 and I use this platform.

There are also a long-term statistics that I really don't know how to use

@SupremeSports
Copy link

For averaging multiple sensors you can use min_max integration.

Yes, this is my first stage of averaging. That way I can have multiple sensors in large rooms (e.g.: garage)

For an average you can use statistics->mean but I already have problems with large averages (+24h) since 2022.4 and I use this platform.

Yes, this is what I switched to for my second stage of averaging in order to eliminate wide variations (e.g.: temp sensors with low resolution). The problem is that it's not possible to average let's say the last 15 seconds to smooth out the value. If the sensors don't change value within that timeframe, then sensor goes unknown and all my dependent sensors, binary sensors, automations, dashboard gauge, climates, everything goes wild.

There are also a long-term statistics that I really don't know how to use

Those seems to be for dashboard history-graph as what I understand and won't be helpful in my case

@rodrigogbs
Copy link

same problem here!

@calielc
Copy link

calielc commented May 5, 2022

same here.

@DavidFW1960
Copy link

DavidFW1960 commented May 6, 2022

check out this one guys... https://github.com/jeroenterheerdt/HADailySensor I think this fixes my requirement for average and min/max

Actually not really as it won't have attributes with min/max. Doh!

@tankdeer
Copy link

tankdeer commented May 6, 2022

check out this one guys... https://github.com/jeroenterheerdt/HADailySensor I think this fixes my requirement for average and min/max

Actually not really as it won't have attributes with min/max. Doh!

I just started using this yesterday and so far it meets my needs. You can do min/max by creating separate sensors for those

@DavidFW1960
Copy link

I just started using this yesterday and so far it meets my needs. You can do min/max by creating separate sensors for those

Yes I know that but I use the flex state card which I can select attributes of an entity to display in a table but I can't specify multiple entities.

@tankdeer
Copy link

tankdeer commented May 6, 2022

I just started using this yesterday and so far it meets my needs. You can do min/max by creating separate sensors for those

Yes I know that but I use the flex state card which I can select attributes of an entity to display in a table but I can't specify multiple entities.

Probably not ideal but could you create a template sensor with all the attributes you need? Might work until you can find a better alternative or this gets fixed

@DavidFW1960
Copy link

I don't think you can create a sensor from a template with attributes. I'm all ears if you have an example

@SupremeSports
Copy link

I don't think you can create a sensor from a template with attributes. I'm all ears if you have an example

Yes you can, I use tons of sensors with attributes... on my phone right now, but the template sensor documentation explains it with examples: https://www.home-assistant.io/integrations/template/

@tankdeer
Copy link

tankdeer commented May 6, 2022

Yup, you definitely can. I'm on my phone, so the formatting might not be perfect, but you just add a map of attributes to the sensor you're creating. Something like this:

- name: My Sensor
  state: >
    {{ some value }}
  attributes:
    some_attribute: >
      {{ attribute value}}
    some_other_attribute: >
      {{ another attribute value}}

Since each attribute is a template, you can populate them with basically any value you want.

@marvin8
Copy link

marvin8 commented May 6, 2022

I i just had a quick look at the profile of @Limych . Says there he is in Moscow, Russia.... means... he might be locked out of GitHub until the situation in the Ukraine has been resolved... :-/

@slimpdev
Copy link

slimpdev commented May 6, 2022

Says there he is in Moscow, Russia.... means...

Me too from Moscow, and no personal sanctions on GitHub. I can use it right before of all this madness.

@VictoriousCupid
Copy link

Quick tip:

Besides the min/max and statistics integrations, there is also "filter"

the combination of these 3 gave me everything I need to replace this dysfunctional integration.

@sshalyminov
Copy link

the combination of these 3 gave me everything I need to replace this dysfunctional integration.

How did you replace "process_undef_as" functionality?
And combination of several entities? (I.e. average of several numeric sensors)

@VictoriousCupid
Copy link

How did you replace "process_undef_as" functionality?

I don't. for me there is no need.

And combination of several entities? (I.e. average of several numeric sensors)

min/max - mean makes an average/mean of multiple entities.
This you can feed into either filter and or statistics.
filter smooths out the output
statistics will make an average over time.

so multiple sensors into min/max. the result put into statistics and/or filter.

come to think of it, you can template for the process_undef_as with:

{% if is_state('sensor.source_abcd', 'unwantwed value 1 like unknown') %} 'wanted value'
{% elif is_state('sensor.source_abcd', 'unwanted value 2 like unavailable') %} 'wanted value'
{% elif is_state('sensor.source_abcd', 'unwanted value 3 like something') %} 'wanted value'
{% else %} {{ (states('sensor.source_abcd') |float ) }} {% endif %}

@lmagyar
Copy link
Contributor

lmagyar commented May 6, 2022

Besides the min/max and statistics integrations, there is also "filter"

The built-in functionality updates the calculated values only when the input sensor changes, so when the input sensor reports the same value for a long time, eg. the average value will remain an old/invalid value and never gets equal with the input sensor. This makes the built-in versions nearly useless (for me). OK, I'm tested this approx. half a year ago, but I don't think this is changed.

@kimmilde
Copy link

kimmilde commented May 6, 2022

I added the get_states back into the init.py and recorder.py files. Everything working now until next time I'll update.. Now I have some time figure out what to do

@matzzzz
Copy link

matzzzz commented May 6, 2022

I added the get_states back into the init.py and recorder.py files. Everything working now until next time I'll update.. Now I have some time figure out what to do

So you forked it and fixed it? anything you want to share?

@kimmilde
Copy link

kimmilde commented May 6, 2022

Yes, no problem! I am using docker, but locate home assistant core folders and go into components/recorder.

In init.py add this at line number 71:

def get_states(hass, utc_point_in_time, entity_ids=None, run=None, filters=None):
    """Return the states at a specific point in time."""
    return history.get_states(
        hass, utc_point_in_time, entity_ids=None, run=None, filters=None
    )


def get_state(hass, utc_point_in_time, entity_id, run=None):
    """Return a state at a specific point in time."""
    return history.get_state(hass, utc_point_in_time, entity_id, run=None)

in history.py add this at line numer 473:

def get_states(
    hass: HomeAssistant,
    utc_point_in_time: datetime,
    entity_ids: list[str] | None = None,
    run: RecorderRuns | None = None,
    filters: Any = None,
    no_attributes: bool = False,
) -> list[State]:
    """Return the states at a specific point in time."""
    if run is None:
        run = recorder.get_instance(hass).run_history.get(utc_point_in_time)

    if run is None or process_timestamp(run.start) > utc_point_in_time:
        # History did not run before utc_point_in_time
        return []

    with session_scope(hass=hass) as session:
        return _get_states_with_session(
            hass, session, utc_point_in_time, entity_ids, run, filters, no_attributes
        )

and this at the bottom:

def get_state(
    hass: HomeAssistant,
    utc_point_in_time: datetime,
    entity_id: str,
    run: RecorderRuns | None = None,
    no_attributes: bool = False,
) -> State | None:
    """Return a state at a specific point in time."""
    states = get_states(hass, utc_point_in_time, [entity_id], run, None, no_attributes)
    return states[0] if states else None

Reboot and you should be good until next time you update

@Murianwind
Copy link
Author

Is there a way to change the custom component's file instead of the HA file?

@ACrazyConcept
Copy link

Wow this is a huge issue for me too if it doesn't work. I hope someone wants to fork it and continue the work. I'll be happy to contribute if I can.

@lmagyar
Copy link
Contributor

lmagyar commented May 6, 2022

Is there a way to change the custom component's file instead of the HA file?

I'm done it: https://github.com/lmagyar/ha-average/commit/acd751587609951a1baf41e6a2b162b0fc0cf3e1
Download and replace the config/custom_components/average/sensor.py with it: https://github.com/lmagyar/ha-average/blob/acd751587609951a1baf41e6a2b162b0fc0cf3e1/custom_components/average/sensor.py

And don't forget to restart HA core!!!

During startup the previously sporadic errors like some sensor is Undefined and cannot be converted to float now changed to Could not convert value "<state sensor.bedroom_temperature=23.4 @ 2022-05-06T13:10:18.153480+02:00>" to float: None is not a recognized temperature unit. But that is an old story.

Disclaimer: I'm just an average HA tinkerer, this is just a workaround, but if it is working for others I will create a PR and somehow "save" this project in HACS.

@rodrigogbs
Copy link

Is there a way to change the custom component's file instead of the HA file?

I'm done it: lmagyar@acd7515 Download and replace the config/custom_components/average/sensor.py with it: https://github.com/lmagyar/ha-average/blob/acd751587609951a1baf41e6a2b162b0fc0cf3e1/custom_components/average/sensor.py

And don't forget to restart HA core!!!

During startup the previously sporadic errors like some sensor is Undefined and cannot be converted to float now changed to Could not convert value "<state sensor.bedroom_temperature=23.4 @ 2022-05-06T13:10:18.153480+02:00>" to float: None is not a recognized temperature unit. But that is an old story.

Disclaimer: I'm just an average HA tinkerer, this is just a workaround, but if it is working for others I will create a PR and somehow "save" this project in HACS.

It worked for me! Thanks @lmagyar

@SupremeSports
Copy link

SupremeSports commented May 6, 2022

I'm done it: lmagyar@acd7515 Download and replace the config/custom_components/average/sensor.py with it: https://github.com/lmagyar/ha-average/blob/acd751587609951a1baf41e6a2b162b0fc0cf3e1/custom_components/average/sensor.py

And don't forget to restart HA core!!!

Works for me too

Disclaimer: I'm just an average HA tinkerer, this is just a workaround, but if it is working for others I will create a PR and somehow "save" this project in HACS.

No, you're great, Thank you @lmagyar...

@alfredros
Copy link

Worked with the fix of @lmagyar
Bravo!

Let's see if the developer can update their code...

@kimmilde
Copy link

kimmilde commented May 6, 2022

Thank you. This made it work again!

@IvoryBalboa
Copy link

With your fix I get an error.

"The average platform for the sensor integration does not support platform setup. Please remove it from your config."

Why is that?

@lmagyar
Copy link
Contributor

lmagyar commented May 6, 2022

"The average platform for the sensor integration does not support platform setup. Please remove it from your config."

I have absolutely no clue. 😲 I'm quite a beginner in HA hacking. 🥺 Though it seems to me some yaml/config error.

@IvoryBalboa
Copy link

I didn't change any config since 2022.5, don't think my yaml is the culprit

@IvoryBalboa
Copy link

For example:
In sensor.yaml:

  • platform: average
    name: 'Messung'
    entities:
    • sensor.testshelly_power
      start: "{{ state_attr('input_datetime.messung_start', 'timestamp') }}"
      end: "{{ state_attr('input_datetime.messung_ende', 'timestamp') }}"

Worked for ages... Now everything is broken 😭

@lmagyar
Copy link
Contributor

lmagyar commented May 6, 2022

Hmmm, I guess something is broken with your sensor.py, this should have been downloaded: https://raw.githubusercontent.com/lmagyar/ha-average/acd751587609951a1baf41e6a2b162b0fc0cf3e1/custom_components/average/sensor.py

@IvoryBalboa
Copy link

Yep, it was broken... Downloaded and copied again, now it works again! 🤩

@Murianwind
Copy link
Author

It's work!! Great!! Thanks a lot!!

@lmagyar
Copy link
Contributor

lmagyar commented May 6, 2022

Just info: If somebody noticed that due to this "transitory" hiccup with this component, the averaged values lost their unit_of_measurement see: #98 (comment)

Note: If you delete data from recorder database, this is your responsibility, only execute that SQL statement if you understand, that all those values are gone forever. And who knows what are the unintended consequences.

@Murianwind
Copy link
Author

There is no problem in execution, but the following log is displayed in the core.

Could this problem be solved?

Logger: homeassistant.helpers.frame
Source: helpers/frame.py:103
First occurred: 11:51:56 (1 occurrences)
Last logged: 11:51:56

Detected integration that accesses the database without the database executor; Use homeassistant.components.recorder.get_instance(hass).async_add_executor_job() for faster database operations. Please report issue to the custom component author for average using this method at custom_components/average/sensor.py, line 477: history_list = history.state_changes_during_period(

@nikipore
Copy link

nikipore commented May 7, 2022

I've been hit by this as well and not willing to live with a hack in my productive system. I am rather proficient in Python, but not in the whereabouts of custom HA integrations – the boilerplate scared me off.

So I implemented a time-weighted rolling average function node with right Riemann sums in Node-RED. Just in case someone else also wants an easy-to-understand and clean solution and uses Node-RED along with HA. It is straightforward to create an averaged HA sensor with an events: state or poll state input node and an entity output node.

[{"id":"c3195a8d84f95c56","type":"function","z":"f2681015255691ca","name":"time-weighted average (over 5s)","func":"/*\ntime-weighted average of payload values\nover the specified time range (in ms)\n*/\n\nlet now = Date.now();\nconst range = 5 * 1000;\n\nlet t = context.get('t') || now - range;\nlet T = context.get('T');\nlet F = context.get('F');\nlet buffer = context.get('buffer');\n\nlet dt = now - t;\nlet increment = {dt: dt, df: Number(msg.payload) * dt};\n\nT += increment.dt;\nF += increment.df;\nbuffer.push(increment);\n\nwhile ((T - buffer[0].dt) >= range) {\n  T -= buffer[0].dt;\n  F -= buffer[0].df;\n  buffer.shift();\n}\n\ncontext.set('t', now);\ncontext.set('T', T);\ncontext.set('F', F);\ncontext.set('buffer', buffer);\n\nmsg.payload = F / T;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"context.set('t', undefined);\ncontext.set('T', 0);\ncontext.set('F', 0);\ncontext.set('buffer', []);","finalize":"","libs":[],"x":490,"y":380,"wires":[["2bd98b77a5b589d1"]]},{"id":"e846aa77ca8fc389","type":"inject","z":"f2681015255691ca","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"0","payloadType":"num","x":270,"y":360,"wires":[["c3195a8d84f95c56"]]},{"id":"c6d55e6e3484ea4e","type":"inject","z":"f2681015255691ca","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"1","payloadType":"num","x":270,"y":400,"wires":[["c3195a8d84f95c56"]]},{"id":"2bd98b77a5b589d1","type":"debug","z":"f2681015255691ca","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":730,"y":380,"wires":[]}]

Start tab:

context.set('t', undefined);
context.set('T', 0);
context.set('F', 0);
context.set('buffer', []);

Function tab:

/*
time-weighted average of payload values
over the specified time range (in ms)
*/

let now = Date.now();
const range = 5 * 1000;

let t = context.get('t') || now - range;
let T = context.get('T');
let F = context.get('F');
let buffer = context.get('buffer');

let dt = now - t;
let increment = {dt: dt, df: Number(msg.payload) * dt};

T += increment.dt;
F += increment.df;
buffer.push(increment);

while ((T - buffer[0].dt) >= range) {
  T -= buffer[0].dt;
  F -= buffer[0].df;
  buffer.shift();
}

context.set('t', now);
context.set('T', T);
context.set('F', F);
context.set('buffer', buffer);

msg.payload = F / T;

return msg;

I chose the right Riemann sum because it behaves nicely with initial values (I assume the initial value has been there for the entire time range) and last values (if the are no frequent updates, the output jumps right to the last value).

@NapalmCandy
Copy link

Is there a way to change the custom component's file instead of the HA file?

I'm done it: lmagyar@acd7515 Download and replace the config/custom_components/average/sensor.py with it: https://github.com/lmagyar/ha-average/blob/acd751587609951a1baf41e6a2b162b0fc0cf3e1/custom_components/average/sensor.py

And don't forget to restart HA core!!!

During startup the previously sporadic errors like some sensor is Undefined and cannot be converted to float now changed to Could not convert value "<state sensor.bedroom_temperature=23.4 @ 2022-05-06T13:10:18.153480+02:00>" to float: None is not a recognized temperature unit. But that is an old story.

Disclaimer: I'm just an average HA tinkerer, this is just a workaround, but if it is working for others I will create a PR and somehow "save" this project in HACS.

It works! Thank you very much!

Contact to developer to do as an official change

@lmagyar
Copy link
Contributor

lmagyar commented May 7, 2022

Detected integration that accesses the database without the database executor; Use homeassistant.components.recorder.get_instance(hass).async_add_executor_job() for faster database operations. Please report issue to the custom component author for average using this method at custom_components/average/sensor.py, line 477: history_list = history.state_changes_during_period(

This comment is a duplicate of #127. FYI: This is unrelated to this fix. And I was using this integration with debug level logging yesterday and haven't seen this warning, strange.

@lmagyar
Copy link
Contributor

lmagyar commented May 7, 2022

Contact to developer to do as an official change

I've contacted him even through email. There is already a ticket to remove this integration from HACS. See: hacs/integration#2654

@lmagyar lmagyar mentioned this issue May 7, 2022
9 tasks
Limych added a commit that referenced this issue May 7, 2022
@DavidFW1960
Copy link

@lmagyar did you see this is going to break again in 2022.6? Bdraco made some notes on how to fix in #127 Have you looked at that as it's all greek to me.

@lmagyar
Copy link
Contributor

lmagyar commented May 11, 2022

Yes, I saw that issue. Sadly it's nearly greek to me also, + I can't reproduce the warning, + I have near zero knowledge about HA internals, + I even don't have a proper development environment. :(

@DavidFW1960
Copy link

Yes, I saw that issue. Sadly it's nearly greek to me also, + I can't reproduce the warning, + I have near zero knowledge about HA internals, + I even don't have a proper development environment. :(

I'm reluctant to reimplement this integration if it's going to be dead next month anyway

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working.
Projects
None yet
Development

Successfully merging a pull request may close this issue.