Filtering out extreme values #57

mark-10-5 · 2021-09-18T21:01:19Z

I seem to be getting some noise on my sensor and I get very extreme values when this happens. These readings are usually over 100,000 W up to over 1 million some times.

I have found a lambda filler that has a max difference value but am not smart enough to combine the current lambda filter with this one and I can't find in the esphome docs if I can have two lambda filters and it process in order or not.

This is the filter I have found
filters:
- lambda: |-
float MAX_DIFFERENCE = 50000.0; // adjust this!
static float last_value = NAN;
if (isnan(last_value) || std::abs(x - last_value) < MAX_DIFFERENCE)
return last_value = x;
else
return {};

And I need to combine it with the current filter of

lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);

lemppari · 2021-09-19T16:35:09Z

You could try this (I haven't tested this...)

        - lambda: |-
            float new_value = x * ((60.0 / ${pulse_rate}) * 1000.0);
            float MAX_DIFFERENCE = 50000.0.0;  // adjust this!
            static float last_value = NAN;
            if (isnan(last_value) || std::abs(new_value - last_value) < MAX_DIFFERENCE)
              last_value = new_value;
            return last_value;

The problem with this is that if the first value is a noise value then the following sane values will be ignored. I think I found the source of your filter example and there is an updated version in that page which should handle that situation a little better. Here is a modified code for that

        - lambda: |-
            float new_value = x * ((60.0 / ${pulse_rate}) * 1000.0);
            float MAX_DIFFERENCE = 50000.0;  // adjust this!
            static float last_value_t = NAN;
            static int count_missed_t = 0;
            if (count_missed_t == 3) last_value_t = new_value;
            if (isnan(last_value_t) || std::abs(new_value - last_value_t) < MAX_DIFFERENCE) {
              count_missed_t = 0;
              last_value_t = new_value;
            } else {
              count_missed_t += 1;
            }
            return last_value_t;

This will handle the first noise value problem. It will revert to new "range of values" after getting three out of range values in a row. Let us know if either of these work.

EDITED: First version compared the values before applying the pulse calculation. Should be fixed now.

mark-10-5 · 2021-09-19T20:06:30Z

I found out by asking on the ESPHome discord that filter values are processed in order of entry in YAML and you can have as many as you like. So i ended up testing it like this last night.

  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 50000.0;  // adjust this!
      static float last_value = NAN;
      if (isnan(last_value) || std::abs(x - last_value) < MAX_DIFFERENCE)
        return last_value = x;
      else
        return {};

I ran the pulse rate calc first as the values i was working with in the graphs in home assistant are after the calculation so I wanted to filter it after the calculation. This worked pretty well, i didn't get any 1 million W readings but i did get 1 96000W reading. So i think your suggestion of the new code is a good one so i am going to test this.

  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 30000.0;  // adjust this!
      static float last_value_t = NAN;
      static int count_missed_t = 0;
      if (count_missed_t == 3) last_value_t = x;
      if (isnan(last_value_t) || std::abs(x - last_value_t) < MAX_DIFFERENCE) {
        count_missed_t = 0;
        return last_value_t = x;
      } else {
        count_missed_t += 1;
        return last_value_t;
      }

I am going with 30000W now as my panel is 125Amp max which is 24000W max draw so anything over 30000W should be well safe as being false.

Thanks for the suggestions and i will report back if this works.

lemppari · 2021-09-20T03:35:28Z

Sorry, I made the comparing to previous values before doing the pulse rate calculation. I modified my code also. Please report your progres. :)

onkelbeh · 2021-09-22T18:22:24Z

Had the same problem, with rare very high spikes (>8kW), solved it with a filter:

sensor:
  - platform: pulse_meter
    name: '${name} - Power consumption'
    unit_of_measurement: 'W'
    state_class: measurement
    device_class: power
    icon: mdi:flash-outline
    accuracy_decimals: 0
    internal_filter: 50ms

I did not use an ESP32, have a NodeMCU12S.
Comparison with the values shown directly on the meter matches.

mark-10-5 · 2021-09-22T23:02:58Z

I still get the odd value with the max difference code. I will add in the internal_filter also.

It has been much better though. I am still trying to pin point where the noise is coming from but not having much luck.

Brink3 · 2021-09-23T12:30:52Z

I also get these spikes, tried your filters but still gets those spkies time to time.
Screenshot from the original code:

.

lemppari · 2021-09-23T16:06:21Z

That is very strange. Do you get only single spikes of crazy values or are there multiple crazy values in row?

If the code from my last example works, you should get four crazy values in a row. Here is a sample what should happen. First row are the new-values calculated in the first line of the code (s prefix is sane value and c is a crazy value). Second row is the output of the filter. Third row is the value of the count_missed_t variable after each step.

s1, s2, c3, s4, s5, c6, c7, c8, c9, s10, s11, s12, s13, s14
s1, s2, s2, s4, s5, s5, s5, s5, c9,  c9,  c9,  c9, s13, s14
0,   0,  1,  0,  0,  1,  2,  3,  0,   1,   2,   3,   0,   0

As we can see there needs to be four crazy values in a row to produce crazy values. And if that happens, even after sane values start coming right after those, the output will be four crazy values in a row before sanes values are the "new normal" again.

Brink3 · 2021-09-23T17:43:15Z

There is just one single crazy spike. I have attached a log from the esphome logging.
Line 4 in this log shows a spike.

[19:21:06][D][light:035]: 'Red' Setting:
[19:21:06][D][light:046]:   State: ON
[19:21:06][D][sensor:121]: 'House - Total energy': Sending state 8.43400 kWh with 3 decimals of accuracy
[19:21:06][D][sensor:121]: 'House - Power consumption': Sending state 45569.61719 W with 0 decimals of accuracy
[19:21:06][D][light:035]: 'Red' Setting:
[19:21:06][D][sensor:121]: 'House - Total energy': Sending state 8.43500 kWh with 3 decimals of accuracy
[19:21:06][D][light:035]: 'Red' Setting:
[19:21:06][D][light:046]:   State: OFF
[19:21:06][D][light:035]: 'Red' Setting:
[19:21:09][D][sensor:121]: 'House - Power consumption': Sending state 1107.69226 W with 0 decimals of accuracy
[19:21:09][D][light:035]: 'Red' Setting:
[19:21:09][D][light:046]:   State: ON
[19:21:09][D][sensor:121]: 'House - Total energy': Sending state 8.43600 kWh with 3 decimals of accuracy
[19:21:10][D][light:035]: 'Red' Setting:
[19:21:10][D][light:046]:   State: OFF
[19:21:12][D][sensor:121]: 'House - Power consumption': Sending state 1116.97168 W with 0 decimals of accuracy
[19:21:12][D][light:035]: 'Red' Setting:
[19:21:12][D][light:046]:   State: ON
[19:21:12][D][sensor:121]: 'House - Total energy': Sending state 8.43700 kWh with 3 decimals of accuracy
[19:21:13][D][light:035]: 'Red' Setting:
[19:21:13][D][light:046]:   State: OFF

lemppari · 2021-09-23T18:42:26Z

Makes no sense to me. You could try to add some logging or take this up with the ESPHome people.

Brink3 · 2021-09-23T19:53:17Z

Makes no sense to me. You could try to add some logging or take this up with the ESPHome people.

Dude, I'm so friggin' stupid. I never changed value of MAX_DIFFERENCE. Changed it now to 40000.0, no spikes so far.

mark-10-5 · 2021-09-23T21:35:44Z

Did you leave it at 50000.0? I think it works better if you calculate the max wattage of your power board and set it to that.

I am finding the internal_filter is working better then the filter code.

Brink3 · 2021-09-23T21:53:12Z

Did you leave it at 50000.0? I think it works better if you calculate the max wattage of your power board and set it to that.

I am finding the internal_filter is working better then the filter code.

Yes, I had it at 50000.0 ;)

Tried the internal_filter @ 50ms and 100ms, but I got same spikes.

mark-10-5 · 2021-09-23T22:22:11Z

Interesting your spikes are always around 40,000, the internal_filter wouldn't catch that as it is not extreme enough. My spikes were much higher, around 100,000+ up to 1,000,000 often and a much wider value range, which the internal filter would catch as it is so extreme so the pulse would be very quick so be filtered out.

40,000 although a high value is a possible value so you will need the max difference calc to filter that one. Glad it seems to be working so far.

JamesSwift · 2021-10-04T14:57:08Z

I'm running on a D1 mini (esp8266) and also getting crazy values at random times.

I wanted to add some thoughts on the above ideas. The internal_filter is handy, but it's essentially a low pass filter. Any crazy value BELOW the internal_filter will still get through. So you still might see a 24,000w crazy value next to a 100w known good value.

The advantage of the lambda filter though, is that you are looking for the DIFFERENCE between the last known good reading and the current reading. However, you still seem to be configuring it based on the maximum possible load of your board, Surely MAX_DIFFERENCE shouldn't be the maximum power draw of your house, but somewhere near the biggest possible load added by a single appliance. For instance, in my house the electric shower is about 40a (around 10kw). The chances of someone turning the shower and another appliance on in the exact same instant is basically zero. So I think a saner value for MAX_DIFFERENCE would be 11000.0.

JamesSwift · 2021-10-05T08:32:44Z

I've managed to solve my crazy values problem!

Firstly, the filter as described above with a value of 11,000.0 worked great at removing the truly crazy values. However now I wasn't distracted by the 6Mw 🤣 pulses, I was still getting random spikes around 1.5kw - 6kw over my base load around 200w.

When you do the math that's a rogue pulse of 2.4s in the midst of a pulse rate of around 18s. That blew my theory that it was just noise in the PWM of the meter's LED. That's a distinct pulse unconnected with the meter's LED. My light detector is sealed off from external light, so what could it possibly be...

Then it struck me! The cable connecting the light detector to the esp8266 ran right next to the radio communications module of the smart meter. It must be an RF induced pulse in the cable! So, I rerouted the cable to be maybe 5cm away and overnight I haven't seen any rogue pulses. It's early days in testing, but I think that was the problem all along. If I still get any interference I'll try adding a 100uf capacitor across the ground and digital output pins as suggested in another issue, which should soak it up.

Just thought it was worth documenting this issue with RF interference here as I'm sure others will encounter it with their smart meter in close proximity,

bictorv · 2021-10-10T14:52:51Z

    internal_filter: 50ms

For me, an internal_filter: 100ms was needed but seems to work very well - no more spikes of hundreds of kilowatt.

EspeeFunsail · 2021-12-11T05:04:48Z

I found out by asking on the ESPHome discord that filter values are processed in order of entry in YAML and you can have as many as you like. So i ended up testing it like this last night.
  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 50000.0;  // adjust this!
      static float last_value = NAN;
      if (isnan(last_value) || std::abs(x - last_value) < MAX_DIFFERENCE)
        return last_value = x;
      else
        return {};
I ran the pulse rate calc first as the values i was working with in the graphs in home assistant are after the calculation so I wanted to filter it after the calculation. This worked pretty well, i didn't get any 1 million W readings but i did get 1 96000W reading. So i think your suggestion of the new code is a good one so i am going to test this.
  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 30000.0;  // adjust this!
      static float last_value_t = NAN;
      static int count_missed_t = 0;
      if (count_missed_t == 3) last_value_t = x;
      if (isnan(last_value_t) || std::abs(x - last_value_t) < MAX_DIFFERENCE) {
        count_missed_t = 0;
        return last_value_t = x;
      } else {
        count_missed_t += 1;
        return last_value_t;
      }
I am going with 30000W now as my panel is 125Amp max which is 24000W max draw so anything over 30000W should be well safe as being false.

Thanks for the suggestions and i will report back if this works.

I tried using this in my own glow.yaml, but it seems like some syntax is wrong? Getting this error:
ERROR Error while reading config: Invalid YAML syntax:

while scanning a simple key
in "/config/esphome/home-assistant-glow.yaml", line 130, column 13:
float new_value = x * ((60.0 / $ ...
^
could not find expected ':'
in "/config/esphome/home-assistant-glow.yaml", line 131, column 13:
float MAX_DIFFERENCE = 4100.0;
^

Anyone able to assist?

EspeeFunsail · 2021-12-11T05:17:15Z

I found out by asking on the ESPHome discord that filter values are processed in order of entry in YAML and you can have as many as you like. So i ended up testing it like this last night.
  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 50000.0;  // adjust this!
      static float last_value = NAN;
      if (isnan(last_value) || std::abs(x - last_value) < MAX_DIFFERENCE)
        return last_value = x;
      else
        return {};
I ran the pulse rate calc first as the values i was working with in the graphs in home assistant are after the calculation so I wanted to filter it after the calculation. This worked pretty well, i didn't get any 1 million W readings but i did get 1 96000W reading. So i think your suggestion of the new code is a good one so i am going to test this.
  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 30000.0;  // adjust this!
      static float last_value_t = NAN;
      static int count_missed_t = 0;
      if (count_missed_t == 3) last_value_t = x;
      if (isnan(last_value_t) || std::abs(x - last_value_t) < MAX_DIFFERENCE) {
        count_missed_t = 0;
        return last_value_t = x;
      } else {
        count_missed_t += 1;
        return last_value_t;
      }
I am going with 30000W now as my panel is 125Amp max which is 24000W max draw so anything over 30000W should be well safe as being false.
Thanks for the suggestions and i will report back if this works.
I tried using this in my own glow.yaml, but it seems like some syntax is wrong? Getting this error: ERROR Error while reading config: Invalid YAML syntax:

while scanning a simple key in "/config/esphome/home-assistant-glow.yaml", line 130, column 13: float new_value = x * ((60.0 / $ ... ^ could not find expected ':' in "/config/esphome/home-assistant-glow.yaml", line 131, column 13: float MAX_DIFFERENCE = 4100.0; ^

Anyone able to assist?

I added some spaces to the stuff below -lamda: and it started to compile, but now I get this:

/data/cache/platformio/packages/framework-arduinoespressif8266/cores/esp8266/Arduino.h:135:16: error: expected unqualified-id before '(' token
#define abs(x) ((x)>0?(x):-(x))
^
/config/esphome/home-assistant-glow.yaml:130:177: note: in expansion of macro 'abs'
float MAX_DIFFERENCE = 4100.0;
^
/config/esphome/home-assistant-glow.yaml:131:3: warning: control reaches end of non-void function [-Wreturn-type]
static float last_value_t = NAN;

Helmasynti · 2021-12-28T07:27:31Z

I had spikes, lots of spikes. Activating the internal filter cut off most insane ones but there was still a lot of noise left. The filter based on difference still improved the situation especially if MAX_DIFFERENCE was set low enough but even this did not solve the problem entirely. I could still see small spikes appearing. The solution was to decrease the sensitivity of the photodiode by turning the potentiometer on the board anti-clockwise. This removed all spikes and I was able to deactivate the filters. The signal is clean now.

If you have noise in the signal from the photodiode, it may be an easy solution to adjust its sensitivity. This can be done especially if you use the given 3D-case, since it blocks very effectively all the interfering light that might come from outside. So the sensitivity can be rather low.

EspeeFunsail · 2021-12-28T08:11:30Z

I had spikes, lots of spikes. Activating the internal filter cut off most insane ones but there was still a lot of noise left. The filter based on difference still improved the situation especially if MAX_DIFFERENCE was set low enough but even this did not solve the problem entirely. I could still see small spikes appearing. The solution was to decrease the sensitivity of the photodiode by turning the potentiometer on the board anti-clockwise. This removed all spikes and I was able to deactivate the filters. The signal is clean now.

If you have noise in the signal from the photodiode, it may be an easy solution to adjust its sensitivity. This can be done especially if you use the given 3D-case, since it blocks very effectively all the interfering light that might come from outside. So the sensitivity can be rather low.

I think this is the issue I'm also having. I will adjust the potentiometer when I get home. The software filters only reduce the values of the spikes, but doesn't actually remove the spikes themselves. I think mine is also just too sensitive.

mcchots · 2022-01-12T12:23:22Z

I added some spaces to the stuff below -lamda: and it started to compile, but now I get this:

/data/cache/platformio/packages/framework-arduinoespressif8266/cores/esp8266/Arduino.h:135:16: error: expected unqualified-id before '(' token #define abs(x) ((x)>0?(x):-(x))
/config/esphome/home-assistant-glow.yaml:130:177: note: in expansion of macro 'abs' float MAX_DIFFERENCE = 4100.0;
/config/esphome/home-assistant-glow.yaml:131:3: warning: control reaches end of non-void function [-Wreturn-type] static float last_value_t = NAN;

I solved this error by changing the functionstd::abs to just abs

matthewharrington · 2022-02-20T05:18:15Z

Hi all. I was having the same issue and came here for assistance. I have tried all sorts of things including internal filter and the extreme value calculator mentioned above.

I was using GPIO12 and have just moved to GPIO26 and all my extreme values have completely disappeared. This is the only thing that worked for me.

Cheers, Matt

louisr0608 · 2022-04-09T19:08:50Z

Hey, just adding my two cents here with another filtering method. The filter I put together is similar to those suggested above, but inverted in a way so that 'large' (1kw) changes have to be confirmed for some number of readings before being accepted. The code is adjustable for how many frames of confirmation you want to have (num_samples). This does skew the readings very slightly, but generally evens out over time -- i.e., if you go up 1500w, it won't register for a few frames, but then when it goes back down the drop won't register for the same number of frames. Its not perfect but it solved all the wacky readings and everything been consistent and reasonably accurate since I added this.

A quick preliminary note, if your sensor has an adjustable sensitivity, make it as sensitive as you can. This filter is also combined with the following settings:

internal_filter_mode: "EDGE"
internal_filter: 35ms

You'll want to adjust internal_filter to be compatible with your meters blink rate (i.e, divide the value by 4 if your meter blinks 4000 x per kw). 35ms works well for mine which is 1k blinks per kw.

Adjust MAX_DIF and num_samples to a desired value to force consecutive readings to register, as described above.


filters:
      - lambda: |-
          const uint num_samples = 3;
          float val = x * ((60.0 / ${pulse_rate}) * 1000.0);
          float ret = val;
          const float MAX_DIF = 1000.0;

          static float ret_z0 = NAN;
          static float samples[num_samples] = {NAN};
          bool inrange;

          if (samples[0] == NAN)
          {
            ret_z0 = val;
            for (int i = 0; i < num_samples; i++)
            {
              samples[i] = val;
            }
          }

          inrange = true;
          for (int i = 0; i < num_samples; i++)
          {
            if ((std::abs(val - samples[i])) > MAX_DIF) inrange = false;
          }

          if (!inrange) ret = ret_z0;

          ret_z0 = ret;
          for (int i = num_samples - 1; i > 0; i--)
          {
            samples[i] = samples[i-1];
          }
          samples[0] = val;

          return ret;

Hope this helps someone!

Regards, Louis

Perka82 · 2022-09-01T16:07:03Z

Changing filer mode to pulse and lower the internal filer to 10 ms solved all my problems. No spikes during the last 72 h.

internal_filter_mode: PULSE
internal_filter: 10ms

filters:
  # multiply value = (60 / imp value) * 1000
  # - multiply: 60
  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);

onkelbeh · 2022-09-03T06:03:45Z

Had the similar problems, also solved it with an internal filter,

sensor:
  - platform: pulse_meter
    name: '${name} - Power consumption'
    unit_of_measurement: 'W'
    state_class: measurement
    device_class: power
    icon: mdi:flash-outline
    accuracy_decimals: 0
    #count_mode:
    #  falling_edge: INCREASE
    #  rising_edge: DISABLE
    internal_filter: 50ms
    pin:
      number: D6
      mode: INPUT
      inverted: true

I have seen no spikes anymore since Nov 21. My measurement is very close to the meter.

martinkantola · 2022-11-20T06:58:22Z

Had the same issue with random extreme spikes. Traced down the issue to the photodiode board itself. The comparator circuit lacks a hysteresis resistor/capacitor. Adding a 33kOhm resistor and 10nF capacitor between pins 5 and 7 of the comparator chip worked for me. Used 1206 size components and soldered them on the adjacent resistors. Also removed the capacitor connected to pin 5 because it was too large for my meter which uses 10.000 pulses/kWh instead of 1000.

martinkantola · 2022-11-20T07:02:28Z

ABredhauer · 2023-01-03T05:16:19Z

I'm also seeing this issue with spikes of 400,000 randomly.

I can see that people are solving this by adding internal filters but I can see different types being used and I'm very new to ESP32 so can someone help me with which is the best approach?

jpaulickcz · 2023-03-13T08:28:04Z

So by default I was getting a lot of crazy spikes (8 000 000 W), however the basic internal_filter: 100ms seems to have sorted 95 % percent of the obviously incorrect readings (after a week of observing), the remaining spikes are now only up to a 10 000-15 000 W. I might play with the more elaborate filter posted by louisr0608 earlier.

I have 500 imp flashes per kWh. I use Wemos D1 Mini with the sensor soldered onto 10 m CAT5e cable, so that might play a role as well.

jessereynolds · 2023-05-22T12:17:29Z

I also had a lot of these crazy spikes going way over what's possible on a single phase residential service (eg 80,000 watts). I have removed the impossible readings with internal_filter: 78ms (theoretical max rate of 60 Amp, 14.4 kW @ 240v, 12.8 pulses per second on my 3200 imp/kWh meter, 78 ms between start of each pulse) and adding a median moving window filter which selects the middle value of the last 3 samples:

    filters:
      - lambda: return x * ((60.0 / id(select_pulse_rate).state) * 1000.0);
      # remove crazy outliers (fingers crossed)
      - median:
          window_size: 3
          send_every: 1
          send_first_at: 1

Going well so far, but I might need to increase the window size, we'll see.

markusg1234 · 2023-10-21T03:58:20Z

Hi,
I was having issues if I opened the power switch board and let light shine in as it would cause a huge spike.
Also some times a spike might occur randomly but not very often.
I am trying a clamp filter and a filter_out and so far so good. I set the max value to 10,000 Watts.

platform: pulse_meter
name: '${friendly_name} - Power Consumption'
id: sensor_energy_pulse_meter
unit_of_measurement: 'W'
state_class: measurement
device_class: power
icon: mdi:flash-outline
accuracy_decimals: 0
pin: ${pulse_pin}
internal_filter_mode: PULSE
internal_filter: 10ms

filters:
  - clamp:
     max_value: 10000
  - filter_out: 10000   
  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);

sparkyws · 2023-12-07T13:41:21Z

Hi all. I was having the same issue and came here for assistance. I have tried all sorts of things including internal filter and the extreme value calculator mentioned above.

I was using GPIO12 and have just moved to GPIO26 and all my extreme values have completely disappeared. This is the only thing that worked for me.

Cheers, Matt

I raised this issue some time ago, the problem with GPIO12/13 is they are touch pins and the wiring will sense this and give incorrect readings, I changed to 26 and it's been working perfectly since

#300

klaasnicolaas · 2023-12-07T13:55:22Z

This weekend I plan to update the repository, if using GPIO26 is really better then I will be happy to adopt that advice in the V4.0.0 release.

Which GPIO should people best use with a Wemos D1 or ESP8266?

sparkyws · 2023-12-07T17:23:56Z

Will depend on the board, but most ESP32's, GPIO13 seem to be touch pins https://lastminuteengineers.com/esp32-pinout-reference/

sparkyws · 2023-12-07T17:26:51Z

The Wemos D1 and the ESP8266 shouldn’t be an issue as they don’t have touch pins

sparkyws · 2023-12-07T19:08:35Z

This weekend I plan to update the repository, if using GPIO26 is really better then I will be happy to adopt that advice in the V4.0.0 release.

I've been running V4.0.0 for 3 months on GPIO26 using an ESP32s with no issues

klaasnicolaas · 2024-01-03T02:37:54Z

FYI: I'm currently working on a refactor of the project where the default pin for the ESP32 boards will from now on be GPIO26.

xsaMNJfkltrwq5r3 · 2024-01-21T06:05:12Z

Before using the software filters it is important to try to get your inputs free of external noise. Try twisting your sensor cables between the sensor and the controller. Then use your software filter to step on anything that gets through. I had the same issue and for me twisting the wires and using a relatively large value on my internal filter time (but less than a realistic pulse period) solved my issue. Recently though I made some changes and I now have the spikes back again, which are occuring at the same time my aircons turn on/off, due to the large surges that would occur at these times. I have to go back and get things back to normal again it will be fine.

bgmn · 2024-03-24T15:55:54Z

Could internal_filter be added to the settings panel just like the Puls rate - imp/kW is now added? That way you could change the internal filter value and still get the updates from github.

klaasnicolaas · 2024-03-24T18:28:07Z

Please create a new issue for that.

./Klaas

xion2000 · 2024-05-06T11:38:02Z

I also had a lot of these crazy spikes going way over what's possible on a single phase residential service (eg 80,000 watts). I have removed the impossible readings with internal_filter: 78ms (theoretical max rate of 60 Amp, 14.4 kW @ 240v, 12.8 pulses per second on my 3200 imp/kWh meter, 78 ms between start of each pulse) and adding a median moving window filter which selects the middle value of the last 3 samples:
    filters:
      - lambda: return x * ((60.0 / id(select_pulse_rate).state) * 1000.0);
      # remove crazy outliers (fingers crossed)
      - median:
          window_size: 3
          send_every: 1
          send_first_at: 1
Going well so far, but I might need to increase the window size, we'll see.

I'm trying and failing to work out the internal filter sizes to set for 2 systems I have setup but simply cannot get my head around the calculation used to work it out and how you can get the 78ms as the answer.

My Glow's are both monitoring meters on solar so the max possible pulse rate will be much lower than a system monitoring grid input.
Both system meters are 1000 imp/kWh and my typical household mains voltage is about 247v.
System 1 max output is approx 1.5kW and system 2 max output is slightly lower at approx 0.8kW.
Can someone please explain the calculation in simply terms.

shytse · 2024-06-20T20:39:38Z

I'm trying and failing to work out the internal filter sizes to set for 2 systems I have setup but simply cannot get my head around the calculation used to work it out and how you can get the 78ms as the answer.

My Glow's are both monitoring meters on solar so the max possible pulse rate will be much lower than a system monitoring grid input. Both system meters are 1000 imp/kWh and my typical household mains voltage is about 247v. System 1 max output is approx 1.5kW and system 2 max output is slightly lower at approx 0.8kW. Can someone please explain the calculation in simply terms.

Let's assume your main breaker flips at 240 V × 50 A, or 12,000 V⋅A. If your meter is rated for 1,000 imp./kWh, at max load it would then pulse at a rate of 12,000 V⋅A × 1,000 imp./kW⋅h = 12kW × 1,000 imp./kW⋅h = 12,000 imp./h, which is equivalent to 12,000/3600 imp./s or about 3.33 times per second. Blinking 3.33 times per second is equivalent to blinking every 1/3.33 s or 300 ms, so any impulse that came earlier than 300ms since previous can be discarded as noise.

jusubbi · 2024-07-13T07:35:48Z

Had the same issue with random extreme spikes. Traced down the issue to the photodiode board itself. The comparator circuit lacks a hysteresis resistor/capacitor. Adding a 33kOhm resistor and 10nF capacitor between pins 5 and 7 of the comparator chip worked for me. Used 1206 size components and soldered them on the adjacent resistors. Also removed the capacitor connected to pin 5 because it was too large for my meter which uses 10.000 pulses/kWh instead of 1000.

I made this modification and it helps to reduce 400kW spikes, but it doesn't remove them all.

I have to install esphome add-on and then make custom firmware to add this line:

sensor:

id: !extend sensor_energy_pulse_meter
internal_filter: 100ms

now there is no spikes any more

klaasnicolaas added the help-wanted We need some extra helping hands or expertise in order to resolve this. label Sep 18, 2021

ABredhauer mentioned this issue Aug 28, 2023

Anyone else having issues with the upgrade to ESPhome 2023.8.1? #301

Closed

klaasnicolaas self-assigned this Jan 3, 2024

Filtering out extreme values #57

Filtering out extreme values #57

Comments

mark-10-5 commented Sep 18, 2021

lemppari commented Sep 19, 2021 • edited Loading

mark-10-5 commented Sep 19, 2021 • edited Loading

lemppari commented Sep 20, 2021 • edited Loading

onkelbeh commented Sep 22, 2021

mark-10-5 commented Sep 22, 2021

Brink3 commented Sep 23, 2021

lemppari commented Sep 23, 2021

Brink3 commented Sep 23, 2021 • edited Loading

lemppari commented Sep 23, 2021

Brink3 commented Sep 23, 2021

mark-10-5 commented Sep 23, 2021

Brink3 commented Sep 23, 2021

mark-10-5 commented Sep 23, 2021 • edited Loading

JamesSwift commented Oct 4, 2021

JamesSwift commented Oct 5, 2021

bictorv commented Oct 10, 2021

EspeeFunsail commented Dec 11, 2021

EspeeFunsail commented Dec 11, 2021 • edited Loading

Helmasynti commented Dec 28, 2021

EspeeFunsail commented Dec 28, 2021

mcchots commented Jan 12, 2022

matthewharrington commented Feb 20, 2022

louisr0608 commented Apr 9, 2022 • edited Loading

Perka82 commented Sep 1, 2022

onkelbeh commented Sep 3, 2022

martinkantola commented Nov 20, 2022

martinkantola commented Nov 20, 2022

ABredhauer commented Jan 3, 2023

jpaulickcz commented Mar 13, 2023

jessereynolds commented May 22, 2023 • edited Loading

markusg1234 commented Oct 21, 2023 • edited Loading

sparkyws commented Dec 7, 2023

klaasnicolaas commented Dec 7, 2023

sparkyws commented Dec 7, 2023 via email • edited Loading

sparkyws commented Dec 7, 2023 via email • edited Loading

sparkyws commented Dec 7, 2023 • edited Loading

klaasnicolaas commented Jan 3, 2024 • edited Loading

xsaMNJfkltrwq5r3 commented Jan 21, 2024 • edited Loading

bgmn commented Mar 24, 2024

klaasnicolaas commented Mar 24, 2024

xion2000 commented May 6, 2024

shytse commented Jun 20, 2024

jusubbi commented Jul 13, 2024 • edited Loading

lemppari commented Sep 19, 2021 •

edited

Loading

mark-10-5 commented Sep 19, 2021 •

edited

Loading

lemppari commented Sep 20, 2021 •

edited

Loading

Brink3 commented Sep 23, 2021 •

edited

Loading

mark-10-5 commented Sep 23, 2021 •

edited

Loading

EspeeFunsail commented Dec 11, 2021 •

edited

Loading

louisr0608 commented Apr 9, 2022 •

edited

Loading

jessereynolds commented May 22, 2023 •

edited

Loading

markusg1234 commented Oct 21, 2023 •

edited

Loading

sparkyws commented Dec 7, 2023 via email •

edited

Loading

sparkyws commented Dec 7, 2023 via email •

edited

Loading

sparkyws commented Dec 7, 2023 •

edited

Loading

klaasnicolaas commented Jan 3, 2024 •

edited

Loading

xsaMNJfkltrwq5r3 commented Jan 21, 2024 •

edited

Loading

jusubbi commented Jul 13, 2024 •

edited

Loading