Daily data resets 50min before 12pm leads to broken graphs, how to filter/replace broken data

[tomatensaus]

I gather data from an inverter, the inverter keeps a daily total of eg. PV production. At 23:09 the clock on the inverter reads 12pm and resets the counter to 0. I have no access to change the date on the inverter. Is there any way I can work around this data problem? I tried to apply filters copied from other examples but the graphs still do not render correctly

This is the history of the sensor

At 23:10 every day all the graphs are not rendered and show 0 values (as the last hour of the day appears as 0)

This problem had me running in circles looking at reasons why data disappears, mean time it works perfectly, just the data source produces broken data with no way to fix it. How do I adjust the filters to discard all the small or 0 values at the end of the day? Below is my yaml copied from bits and pieces I found in other examples (with some filters I tried to fix the issue)

type: custom:plotly-graph
view_layout:
  grid-area: daily
entities:
  - entity: sensor.deyeinvertermaster_summary_day_pv
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Solar" + "🔆" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(255, 155, 48)
    filters:
      - filter: i > 0 && i < (xs.length - 1)
      - fn: |
          ({ys,xs,hass}) => ({
            xs: [...xs, new Date().setHours(0,0,0)],
            ys: [...ys, hass.states['sensor.deyeinvertermaster_summary_day_pv'].state],
          })
  - entity: sensor.deyeinvertermaster_summary_day_load
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Load" + "⚡" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(95, 182, 173)
    filters:
      - filter: i > 0 && i < (xs.length - 1)
      - fn: |
          ({ys,xs,hass}) => ({
            xs: [...xs, new Date().setHours(0,0,0)],
            ys: [...ys, hass.states['sensor.deyeinvertermaster_summary_day_load'].state],
          })
  - entity: sensor.deyeinvertermaster_summary_day_grid_importbuy
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Grid Import" + "💡" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(84, 144, 194)
    filters:
      - filter: i>0
      - force_numeric
  - entity: sensor.deyeinvertermaster_summary_day_grid_exportsell
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Grid Export" + "💡" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(244, 44, 94)
    filters:
      - filter: i>0
  - entity: sensor.deyeinvertermaster_summary_day_battery_discharge
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Battery Discharge" + "🖱️" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(151, 90, 182)
    filters:
      - filter: i>0
      - force_numeric
  - entity: sensor.deyeinvertermaster_summary_day_battery_charge
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Battery Charge" + "🔋" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    marker:
      color: yellow
    filters:
      - filter: i>0
      - force_numeric
hours_to_show: 5d
title: Daily Performance
refresh_interval: 10
defaults:
  yaxes:
    fixedrange: true
layout:
  legend:
    bgcolor: rgba(0,0,0,0)
    itemsizing: constant
    font:
      size: 11
  height: 410
config:
  displayModeBar: false
  scrollZoom: true

[dbuezas]

Some ideas:

• Use hourly statistics and do the daily aggregation yourself inside a filter
• Create a new template entity in Home Assistant that correct this
• Try statistic: max instead of state

[tomatensaus]

type: custom:plotly-graph
view_layout:
  grid-area: daily
entities:
  - entity: sensor.deyeinvertermaster_summary_day_pv
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Solar" + "🔆" + "(" +ys[ys.length - 1]+"kWh)"
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(255, 155, 48)
    width: $fn() => 1000*60*60*3
    period: hour
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
  - entity: sensor.deyeinvertermaster_summary_day_load
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Load" + "⚡" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    width: $fn() => 1000*60*60*3
    marker:
      color: rgb(95, 182, 173)
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
  - entity: sensor.deyeinvertermaster_summary_day_grid_importbuy
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Grid Import" + "💡" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(84, 144, 194)
    width: $fn() => 1000*60*60*3
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
  - entity: sensor.deyeinvertermaster_summary_day_grid_exportsell
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Grid Export" + "💡" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(244, 44, 94)
    width: $fn() => 1000*60*60*3
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
  - entity: sensor.deyeinvertermaster_summary_day_battery_discharge
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Battery Discharge" + "🖱️" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(151, 90, 182)
    width: $fn() => 1000*60*60*3
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 12h
  - entity: sensor.deyeinvertermaster_summary_day_battery_charge
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Battery Charge" + "🔋" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    marker:
      color: yellow
    width: $fn() => 1000*60*60*3
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
hours_to_show: 5d
title: Daily Performance
refresh_interval: 10
defaults:
  yaxes:
    fixedrange: true
layout:
  legend:
    bgcolor: rgba(0,0,0,0)
    itemsizing: constant
    font:
      size: 11
  height: 410
config:
  displayModeBar: false
  scrollZoom: true

I tried with the max... I also had to map_x: +x + 2*60*60*1000
If I map with 1 hour the graph does not show at all, some of today's values not rendering. I can see some of the PV graphs showing, but other values not showing at all

[tomatensaus]

Spotted my mistake... some of the periods remained as "day" should be hours, graph looking much better

bars are overlapping, why does this happen? Easiest way to fix them?

[tomatensaus]

I added some manual offsets

type: custom:plotly-graph
view_layout:
  grid-area: daily
entities:
  - entity: sensor.deyeinvertermaster_summary_day_pv
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Solar" + "🔆" + "(" +ys[ys.length - 1]+"kWh)"
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(255, 155, 48)
    width: $fn() => 1000*60*60*3
    offset: $fn() => 1000*60*60*1
    period: hour
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
  - entity: sensor.deyeinvertermaster_summary_day_load
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Load" + "⚡" + "(" +ys[ys.length - 1]+"kWh)"
    period: hour
    type: bar
    texttemplate: '%{y}'
    width: $fn() => 1000*60*60*3
    offset: $fn() => 1000*60*60*5
    marker:
      color: rgb(95, 182, 173)
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
  - entity: sensor.deyeinvertermaster_summary_day_grid_importbuy
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Grid Import" + "💡" + "(" +ys[ys.length - 1]+"kWh)"
    period: hour
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(84, 144, 194)
    width: $fn() => 1000*60*60*3
    offset: $fn() => 1000*60*60*9
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
  - entity: sensor.deyeinvertermaster_summary_day_grid_exportsell
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Grid Export" + "💡" + "(" +ys[ys.length - 1]+"kWh)"
    period: hour
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(244, 44, 94)
    width: $fn() => 1000*60*60*3
    offset: $fn() => 1000*60*60*13
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
  - entity: sensor.deyeinvertermaster_summary_day_battery_discharge
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Battery Discharge" + "🖱️" + "(" +ys[ys.length - 1]+"kWh)"
    period: hour
    type: bar
    texttemplate: '%{y}'
    marker:
      color: rgb(151, 90, 182)
    width: $fn() => 1000*60*60*3
    offset: $fn() => 1000*60*60*17
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
  - entity: sensor.deyeinvertermaster_summary_day_battery_charge
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Battery Charge" + "🔋" + "(" +ys[ys.length - 1]+"kWh)"
    period: hour
    type: bar
    texttemplate: '%{y}'
    marker:
      color: yellow
    width: $fn() => 1000*60*60*3
    offset: $fn() => 1000*60*60*21
    filters:
      - fn: |
          ({xs, ys}) => {
            let max = 0;
            let curr_day = new Date(xs[0]).setHours(0,0,0,0);
            const ys2 = ys.map((y,i)=> {
              if (isNaN(y)) return y;
              const day = new Date(xs[i]).setHours(0,0,0,0);
              console.log(curr_day,day);
              if (day !== curr_day) {
                curr_day = day; 
                max = y;
              }
              max = Math.max(max, y);
              return max;
            }, max);
            console.log(ys2)
            return { ys: ys2 };
          }    
      - map_x: +x + 2*60*60*1000
      - resample: 1d
hours_to_show: 5d
title: Daily Performance
refresh_interval: 10
defaults:
  yaxes:
    fixedrange: true
layout:
  legend:
    bgcolor: rgba(0,0,0,0)
    itemsizing: constant
    font:
      size: 11
  height: 410
config:
  displayModeBar: false
  scrollZoom: true

[tomatensaus]

I realised that data reported was wrong above, somehow all data was reported for +2 hour offset which I think was related to my timezone. To be honest I have given up. I simply cannot figure out how data gets returned and how to slice it up in proper days and report it in a meaningful day for Days and Months. The graph above looks amazing but the data is wrong.
The moment there is some missing data all falls apart and the graph does not render at all.

[rdpeake]

I know this is after the fact - but i was able to resolve the weird data issues by using hour in combination with the following filters:

filters:
  - fn: |
      ({xs, ys}) => {
        let xs2 = xs.map((x, i) => new Date(x).setHours(0,0,0,0))
          .filter((value, index, array) => array.indexOf(value) === index);
        let max = 0;
        let curr_day = xs2[0];
        let ys2 = xs2.map((x) => ys.reduce((ac, val, i) => {
          const day = new Date(xs[i]).setHours(0,0,0,0);
          if (day == x) {
            return Math.max(ac, val)
          }
          return ac
        }, 0));
        console.log(xs2, ys2)
        return { xs: xs2, ys: ys2 };
      }

this stripped the data into a dataset of "date" and "max value", so you weren't needing to use any remappings of the x values via alternative filters.

[tomatensaus]

I was able to get someone to apply the correct date on the inverter but I would be interested to know if I can work around the broken data that was recorded in statistics

This is how it currently renders with the above code

Using statistic: max breaks all days except today in the graph below. I applied Max to the orange and blue graph, (PV Load and Grid)

I also marked the area where the data is broken vs days that have correct data but not showing with statistic:max