Refactor backfill for unit testing #1214
Conversation
… Mongo collection from the database is returned. -- I investigated the SOSA Mongo module that db.trades and db.resume_markers wrapped. I couldn't determine a real benefit to SOSA, as opposed to going directly to the Mongo DB object. SOSA seems to be one abstraction too many. If I am wrong, it is easy enough to add it back. For now, simple wins.
…t-one test/_specs/commands/backfill/backfill.process.function.test.js' for example. To run all tests 'npm test'.
|
GDAX is the only exchange that seems to offer a reliable backfill source. In my testing, the other exchanges, fail in some way or another. If you have access to other exchanges, please test this PR, and let me know if they are working as expected for you. Thanks! |
|
|
||
| if (data !== undefined && typeof process.stdout.clearLine == 'function') { | ||
| process.stdout.clearLine(); | ||
| process.stdout.write("Processed trades up to " + Moment(data.time).fromNow() + "." ); |
There was a problem hiding this comment.
do you think we can show here exactly how many trades we have processed? I think that might give a better sense of what the bot is exactly doing (and what is taking so long)
There was a problem hiding this comment.
Good call.. I will make that change soon. I was going to have it display the time that the last trades were received/processed, too.
|
My earlier comment is collapsed above, but just so its out there, I added the bit about showing the number of trades processed during backfilling in 5261f02 |
|
My tests look good 👍 merging |
|
fyi this merge breaks backfill in some way and so also live trading warmup: new: old: there is something wring in starttime and endtime calcualtion. noticed via binance exchange, but possible an overall problem? |
|
I am also seeing this one breaking backfill as well. running backfill on gdax takes a very long time. When i use the trade option and it backfills the recent trade it pretty much sits saying backfilling.... |
|
because it starts backfiling somewhere around 1970 :) |
|
Noted... I will look into the issues tonight. |
|
see #1271 |
|
See #1279. |
* If ObjectifySelector is passed an object, it will return it, unmodified. * Removed calls to db.trades and db.resume_markers. Instead, the direct Mongo collection from the database is returned. -- I investigated the SOSA Mongo module that db.trades and db.resume_markers wrapped. I couldn't determine a real benefit to SOSA, as opposed to going directly to the Mongo DB object. SOSA seems to be one abstraction too many. If I am wrong, it is easy enough to add it back. For now, simple wins. * Added scripts for testing. To call a single test, 'npm run-script test-one test/_specs/commands/backfill/backfill.process.function.test.js' for example. To run all tests 'npm test'. * Refactored backfill for unit testability. * Added check to see if exchange offers backfill data to backfill.js * Only flush resumeMarkers if there is at least one existing. * Minor, check for debugging before printing status * ResumeMarkerService calls the callback when there are no ranges, and flush() is called. * Removed debugger statement * Now displaying a counter of the number of trades backfilled in the current session.
No functionality change, other than UI, and that is minor. The code has been rewritten with an eye toward DRY, SOLID, code. Take note of the
Please test test test!