Fixing spatial anomaly for htmcore WIP #15

breznak · 2019-07-31T21:13:02Z

So, we have a problem.. hopefully a combination of the params for HTMcore is not optimal.

The Numenta detectors have an "artificial" way of detecting "spatial anomaly" (which is a running upper/lower bounds threshold detector)

If I disable such (fake) detector for our htmcore the scores become pretty bad:

    "htmcore": {
-        "reward_low_FN_rate": 66.1922106936328,
-       "reward_low_FP_rate": 58.7930712907694,
-        "standard": 63.081419488725054
+        "reward_low_FN_rate": 28.85203529943721,
+       "reward_low_FP_rate": 23.591809889328452,
+       "standard": 26.783416390999154

This means most of the work in htmcore with these settings is done in the threshold detector.

Luckily, the numenta detectors & settings do not suffer this drop when spatial-threshold is removed.

TL;DR: we must optimize our net without this "spatial anomaly" threshold-detector.

For #6

breznak · 2019-07-31T21:13:17Z

CC @ctrl-z-9000-times

breznak · 2019-08-01T08:13:00Z

One interesting fact,
using synapse competition from htm-community/htm.core#584
leads to better utilization of synapses (probably counters some ill combination of params) which leads to our results getting back to reasonable again.

HTMcore without "spatial anomaly" and with "synapse competition":

--- a/results/final_results.json
+++ b/results/final_results.json
@@ -15,9 +15,9 @@
"standard": 16.43666922426724
},
"htmcore": {
-        "reward_low_FN_rate": 66.1922106936328,
-        "reward_low_FP_rate": 58.7930712907694,
-        "standard": 63.081419488725054
+        "reward_low_FN_rate": 59.48811919632727,
+        "reward_low_FP_rate": 53.995064634304576,
+        "standard": 56.90459258759435
},

EDIT:
Ok, scratch that. I apparently ran NAB on an incorrect branch, the results are still bad w/ syn_competition and w/o spatial anomaly.

ctrl-z-9000-times · 2019-08-01T13:48:48Z

Yea I don't know what to say here. Numenta did several non-biological things in order score higher on the NAB benchmark.

I think we should disregard both the spatial anomaly and the backtracking TM, since both are non-biological methods which improve scores on this particular benchmark (but probably not all benchmarks).

breznak · 2019-08-01T14:16:30Z

Numenta did several non-biological things in order score higher on the NAB benchmark. ..backtrackingTM

yep, we're catching up to "HTM" with backTM, even Numenta lists the numentaTM detecot, which uses biological TM and our htmcore detector has results close to that one:
htm-community/htm.core#391

disregard both the spatial anomaly

definitely we should. I was really worried even Numenta's detector suffer the problem (most work done by "spatial anomaly"), fortunately it is not a problem (so the NAB paper is valid!), unfortunately, our TM suffers w/o the spatial anomaly.
For param optimization we must not use the "spatial anomaly". I'll adapt the PR not-to use it on our detector (which is the correct approach). Not sure what to do with Numenta's, probably leave as is?

This reverts commit 354a025.

which is "fake" and non-biological, not HTM related. Unfortunately, with the current settings the code affected the results the most.

breznak · 2019-10-04T12:26:47Z

@ctrl-z-9000-times I'll try to revisit this again, the issue is a weird/severe performance regression. Please help me investigating if you have time, thanks

This reverts commit 09f2839.

breznak · 2020-04-16T13:48:22Z

Update:
When running on only the artificial/synthetic labels, our results are quite good:
python run.py -d htmcore --detect --score --optimize --normalize --windowsFile labels/synthetic.json -n 8

htmcore detector benchmark scores written to /mnt/store/devel/HTM/NAB/results/htmcore/htmcore_reward_low_FN_rate_scores.csv

Running score normalization step
Final score for 'htmcore' detector on 'standard' profile = 84.84
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 84.36
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 88.37
Final scores have been written to /mnt/store/devel/HTM/NAB/results/final_results.json.

This reverts commit e34e6f4.

Zbysekz · 2020-09-15T06:09:38Z

I merge it and i got

Running score normalization step
Final score for 'htmcore' detector on 'standard' profile = 66.35
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 57.11
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 71.82
Final scores have been written to /home/ub/HTM/NAB/results/final_results.json.

It is the "good" score?

I can merge it with this branch if you like
EDIT: i will commit it there

Zbysekz · 2020-09-15T06:30:51Z

I don't have write permissions for this NAB repo.

The branch is there: https://github.com/Zbysekz/NAB/tree/fixing_spatial_anomaly

Zbysekz · 2020-09-15T06:31:57Z

Now i got even better: (assume it is because of seed==0)

Final score for 'htmcore' detector on 'standard' profile = 70.51
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 60.67
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 75.74

Zbysekz · 2020-09-15T06:41:32Z

Previous was with spatial anomaly=True,

with false it is:

Final score for 'htmcore' detector on 'standard' profile = 60.91
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 52.14
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 65.60

Not sure if it should be set to true or false in this branch?

breznak · 2020-09-15T08:25:53Z

OK, i take a look at this. Panda remake is now "finished", so now it uses NetworkAPI only.

thank you 👍
ad NetworkAPI: Does it mean we won't be able to use it with this detector? Since the detector is not in NAPI.

It is the "good" score?

that was slightly below good :), I think I got same-as Numenta scores. (hope with correct settings too)
But I did use the current htmcore master build, not the latest tag from pip (there's the difference in improving changes).

Not sure if it should be set to true or false in this branch?

we want spatial=False here, and everywhere.

Fixing spatial anomaly

breznak · 2020-09-15T08:34:03Z

@Zbysekz btw, the htmcore detector was not in your PR, so the conflict is still there

Zbysekz · 2020-09-15T08:34:43Z

ad NetworkAPI: Does it mean we won't be able to use it with this detector? Since the detector is not in NAPI.

Yes, it would be not possible anymore. I coudn't keep Panda in this way. With using NAPI, everything is more general and generated automatically in contrast of manually handing over specific data & instances.

But i can rewrite this nab detector to use NAPI, what i saw it should be quite simple.

Zbysekz · 2020-09-15T08:38:21Z

btw, the htmcore detector was not in your PR, so the conflict is still there

Hmm there is a bit complication, because i wanted to revert pandaVis changes, but these are not in this branch yet so that is why it was not reflected... i will start new PR and we will just link it from here

breznak · 2020-09-15T09:01:04Z

But i can rewrite this nab detector to use NAPI, what i saw it should be quite simple.

I was thinking about it...let's keep it for a while as is, and if we stabilize the (top) scores, we can rewrite to NAPI. What I wonder is overhead of the NAPI, for c++->py it's about 30%.

Zbysekz · 2020-09-15T09:39:31Z

What I wonder is overhead of the NAPI, for c++->py it's about 30%.

What you mean by this overhead of 30% ?
Like performance overhead/complexity or line count?

breznak · 2020-09-15T09:54:57Z

What you mean by this overhead of 30% ?

compute speed overhead.

Undo panda vis and resolve merge conflict in fix_spatial

Zbysekz · 2020-09-15T10:08:58Z

What you mean by this overhead of 30% ?

compute speed overhead.

Hmm yes 30% overhead is good, i would expect more.

Zbysekz · 2020-09-15T10:40:07Z

I am getting

Final score for 'htmcore' detector on 'standard' profile = 60.91
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 52.14
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 65.60

without spatial anomaly and with latest htm.core

gotham29 · 2022-10-19T17:19:46Z

I am getting

Final score for 'htmcore' detector on 'standard' profile = 60.91
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 52.14
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 65.60

without spatial anomaly and with latest htm.core

Hi @Zbysekz!

Could I possibly see your implementation at all?
I ask because I'm getting NAB score around 50 with my htm.core implmenetation (htm_streamer), and itching to figure out the difference.
I know its been a while now but just wondering!

Thanks,

Sam

gotham29 · 2022-10-19T17:23:41Z

Scores with random (seed=0), without any further optimizations

 "htmcore": {
-        "reward_low_FN_rate": 76.5626293570994,
-        "reward_low_FP_rate": 61.359926511549155,
-        "standard": 71.3094612770284
+        "reward_low_FN_rate": 73.96624112193469,
+        "reward_low_FP_rate": 59.253001477255935,
+        "standard": 68.40111896667753

Hey @breznak!

Could I possibly see your implementation that got these scores if you still have it??
I am trying to validate my own htm.core implementation (htm_streamer) on NAB and only scoring around 50 - and dying to get to the bottom of it.
Thank you!

Sam

Zbysekz · 2022-10-20T02:29:55Z

I am getting
Final score for 'htmcore' detector on 'standard' profile = 60.91
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 52.14
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 65.60
without spatial anomaly and with latest htm.core
Hi @Zbysekz!

Could I possibly see your implementation at all? I ask because I'm getting NAB score around 50 with my htm.core implmenetation (htm_streamer), and itching to figure out the difference. I know its been a while now but just wondering!

Thanks,

Sam

Hello Sam,
I was just debugging connection with pandaVis, i was not improving the NAB scoring. So my setup was just this branch compiled.

What about this? "good as Numenta results without the fake spatial anomalies" as Breznak writes in one comment. I thnik we discuss this somewhere, but it is already so long ago. I am not i this project active right now.
Trying to remember, but i think there was some mathematic simplification for calculating spatial anomalies.

Good luck

gotham29 · 2022-10-20T02:34:24Z

Hi @Zbysekz!
Thanks for your reply!
Do you mean that you you might've added the spatial anomaly functionality to your htm.core implementation to score > 60?
I see in the NumentaTM detector they declare spatial anomaly for anything more than 5% above the max or below the min.

Zbysekz · 2022-10-21T05:22:31Z

Hi @Zbysekz! Thanks for your reply! Do you mean that you you might've added the spatial anomaly functionality to your htm.core implementation to score > 60? I see in the NumentaTM detector they declare spatial anomaly for anything more than 5% above the max or below the min.

No i just ran latest htm.core with this branch (that is not merged to NAB master) with spatial anomaly off..
I wish i could help more.

gotham29 · 2022-10-21T05:25:27Z

Hi @Zbysekz! Thanks for your reply! Do you mean that you you might've added the spatial anomaly functionality to your htm.core implementation to score > 60? I see in the NumentaTM detector they declare spatial anomaly for anything more than 5% above the max or below the min.

No i just ran latest htm.core with this branch (that is not merged to NAB master) with spatial anomaly off.. I wish i could help more.

All good man thank you!
I now have 60+ NAB score when using spatial anomaly, which is comparable to NuPIC & htm.java so I'm happy!
With that done I'm definitely curious to look into your visualization functionality too!

ctrl-z-9000-times · 2022-10-21T14:25:37Z

Hi @Zbysekz and @gotham29,
If you're interested in working on NAB, I will give you write permissions to this repo.

gotham29 · 2022-10-21T16:11:59Z

Hi @ctrl-z-9000-times, yes thank you that'd be awesome!
I'd love to have a look!

ctrl-z-9000-times · 2022-10-25T01:45:43Z

Ok @gotham29, you should now have write permission for this repo.
Feel free to improve this project in whatever way you see fit.

Disabled spatial anomaly leads to terrible score for HTMcore

354a025

breznak added the question Further information is requested label Jul 31, 2019

breznak mentioned this pull request Aug 1, 2019

Spatial anomaly htm-community/htm.core#587

Open

5 tasks

breznak added the high high importance issue label Aug 1, 2019

This was referenced Aug 1, 2019

Connections: apply synapsesCompetition() in adaptSegment() htm-community/htm.core#584

Open

Improve Anomaly performance on NAB in TM to match BacktrackingTM htm-community/htm.core#391

Open

breznak added 2 commits August 3, 2019 15:13

Revert "Disabled spatial anomaly leads to terrible score for HTMcore"

a51f60d

This reverts commit 354a025.

Htmcore detector: remove artificial spatial-anomaly code

d6023e3

which is "fake" and non-biological, not HTM related. Unfortunately, with the current settings the code affected the results the most.

breznak force-pushed the fixing_spatial_anomaly branch from c40c217 to d6023e3 Compare August 3, 2019 14:18

breznak added 4 commits August 3, 2019 19:41

WIP tuning score

3a0c513

htmcore: added date-time encoder fields

ae2e699

htmcore: tune params 2

4fe8b48

tuning SP

387b744

breznak self-assigned this Oct 4, 2019

breznak added 4 commits October 4, 2019 14:28

Merge branch 'master' into fixing_spatial_anomaly

34d369d

updating configs

09f2839

Revert "updating configs"

1767a98

This reverts commit 09f2839.

improve import

353fc81

This was referenced Feb 4, 2020

Upstream numenta/NAB now supports py3, how to merge? #21

Open

Implement Naive, SpatialAnomaly detectors for (numenta)/NAB #22

Open

psteinroe mentioned this pull request Apr 13, 2020

Performance Comparison to nupic htm-community/htm.core#792

Open

breznak added 2 commits April 16, 2020 14:59

Merge branch 'master_community' into fixing_spatial_anomaly

c93715e

Merge branch 'master_community' into fixing_spatial_anomaly

eaa5828

Merge branch 'master_community' into fixing_spatial_anomaly

ce9d08e

Revert "Merge pull request #33 from Zbysekz/pandaVisImplementation"

0cc8545

This reverts commit e34e6f4.

Merge branch 'undoPandaVis' into fixing_spatial_anomaly

7ee95cd

Merge pull request #42 from Zbysekz/fixing_spatial_anomaly

d72fbe3

Fixing spatial anomaly

Zbysekz added 2 commits September 15, 2020 10:46

undo pandaVis implementation

9ef15d4

Merge branch 'fixing_spatial_anomaly' into undo_PandaVis

4d4e594

Zbysekz mentioned this pull request Sep 15, 2020

Undo panda vis and merge fix spatial #43

Closed

Not using spatial anomaly

755027e

Merge pull request #44 from Zbysekz/undo_PandaVis_and_mergeFixSpatial

e835f88

Undo panda vis and resolve merge conflict in fix_spatial

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixing spatial anomaly for htmcore WIP #15

Fixing spatial anomaly for htmcore WIP #15

breznak commented Jul 31, 2019 •

edited

Loading

breznak commented Jul 31, 2019

breznak commented Aug 1, 2019 •

edited

Loading

ctrl-z-9000-times commented Aug 1, 2019

breznak commented Aug 1, 2019

breznak commented Oct 4, 2019

breznak commented Apr 16, 2020

Zbysekz commented Sep 15, 2020 •

edited

Loading

Zbysekz commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

breznak commented Sep 15, 2020

breznak commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

breznak commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

breznak commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

gotham29 commented Oct 19, 2022

gotham29 commented Oct 19, 2022

Zbysekz commented Oct 20, 2022

gotham29 commented Oct 20, 2022

Zbysekz commented Oct 21, 2022

gotham29 commented Oct 21, 2022

ctrl-z-9000-times commented Oct 21, 2022

gotham29 commented Oct 21, 2022

ctrl-z-9000-times commented Oct 25, 2022

Fixing spatial anomaly for htmcore WIP #15

Are you sure you want to change the base?

Fixing spatial anomaly for htmcore WIP #15

Conversation

breznak commented Jul 31, 2019 • edited Loading

breznak commented Jul 31, 2019

breznak commented Aug 1, 2019 • edited Loading

ctrl-z-9000-times commented Aug 1, 2019

breznak commented Aug 1, 2019

breznak commented Oct 4, 2019

breznak commented Apr 16, 2020

Zbysekz commented Sep 15, 2020 • edited Loading

Zbysekz commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

breznak commented Sep 15, 2020

breznak commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

breznak commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

breznak commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

Zbysekz commented Sep 15, 2020

gotham29 commented Oct 19, 2022

gotham29 commented Oct 19, 2022

Zbysekz commented Oct 20, 2022

gotham29 commented Oct 20, 2022

Zbysekz commented Oct 21, 2022

gotham29 commented Oct 21, 2022

ctrl-z-9000-times commented Oct 21, 2022

gotham29 commented Oct 21, 2022

ctrl-z-9000-times commented Oct 25, 2022

breznak commented Jul 31, 2019 •

edited

Loading

breznak commented Aug 1, 2019 •

edited

Loading

Zbysekz commented Sep 15, 2020 •

edited

Loading