Performance benchmark for Connections #490
Conversation
chetan51
changed the title
|
@chetan51 Great to see some initial progress. One thought on this: what if we trained general temporal memory on the full hotgym dataset (or ideally an even longer dataset) and saved the connections datastructure. One of the benchmarks could simply load it and then use the various API calls on that structure in a long loop. ?? |
|
@subutai That's a good idea for one of the tests. We would also want to exercise the actual process of learning from scratch as well, maybe in separate tests. |
|
Also we might be able to use numenta/nupic-legacy#2066 to do the same except with the Spatial Pooler. |
| # Setup test_connections_performance | ||
| # | ||
| set(EXECUTABLE_CONNECTIONSPERFORMANCETEST connections_performance_test) | ||
| add_executable(${EXECUTABLE_CONNECTIONSPERFORMANCETEST} test/integration/ConnectionsPerformanceTest.cpp) |
|
If the tests are all in one file then no need for the header. How big is the resulting binary? It should be pretty small. |
|
@scottpurdy It's 552K. I kept a separate header to keep consistent with the other tests in nupic.core. Do you want me to remove it? |
|
@chetan51 - yeah I would say to remove it. This isn't a normal test, it is a stand-alone executable. Plus, our tests should get moved over to using the gtest framework in which you don't have headers for the tests. Edit: Actually it might be good to keep the header in case someone else wants to create an executable wrapping these tests. |
|
👍 |
chetan51
changed the title
|
@scottpurdy Implemented the spatial pooler test, still have to implement the temporal pooler test. Please review. |
| virtual void RunTests(); | ||
|
|
||
| private: | ||
| void testTemporalMemoryUsage(); |
There was a problem hiding this comment.
The point of having a header is so that someone can create their own program that wraps this. So I would make the test methods public in case the caller doesn't want to run all of them, which is likely the case when profiling.
|
Added Temporal Pooler test, and modified SP test. Unfortunately the TP test is segfaulting for some reason, so I'll need to debug that now. |
|
@scottpurdy Fixed the TP test. The following 4 tests are implemented: spatial pooler, temporal memory, large temporal memory, temporal pooler. This completes the benchmark. This PR is ready for review now. |
|
Can you move the different test cases to different PRs? Just easier to review. |
|
@scottpurdy It will be hard because they share common functions. Here are the four test cases: https://github.com/numenta/nupic.core/pull/490/files#diff-84cb4b1a31bb08de73d252fe165e4033R55 And here are the shared functions: https://github.com/numenta/nupic.core/pull/490/files#diff-84cb4b1a31bb08de73d252fe165e4033R84 If you still need me to split up the PR, let me know and I can do that. |
chetan51
changed the title
|
One question about headers still outstanding and a few lines over 80 characters but otherwise looks good |
|
@scottpurdy Fixed lines over 80 characters. Ready to merge? |
|
One last suggestion: https://github.com/numenta/nupic.core/pull/490/files#r35378766 |
|
@scottpurdy Done. |
|
You need to add the Connections Additionally, don't use the |
|
@scottpurdy Is that better? |
|
@scottpurdy Updated. |
|
👍 but did this cause the AppVeyor failure? |
|
I can't tell why the AppVeyor build failed. @rhyolight can you tell? |
Performance benchmark for Connections
|
Excellent. I'm looking at the divergence in the extensive tests, but had some RL distractions to deal with.. |
Finish Cereal serialization
Fixes #491.
Usage:
make tests_connections_performance