Suggestions for use of py65 with testing suite?

[scotws]

Because somebody here said "testing" (see #39 ), I thought I'd mention that over at Tali Forth, we've started using a test suite that feeds lines from a Forth script to a program (Tali Forth) running on the py65 and checks the answers (https://github.com/scotws/TaliForth2/tree/master/tests).This is very much still work in progress (see the issues related to testing), and though it works, I'm wondering if this is the best way to do it or if you have any suggestions, because we're probably not the first people to try something like this. Thanks!

[BigEd]

I like the idea, but I've no ideas to contribute...

[mnaberez]

Looks like the link above got mangled, it's:
https://github.com/scotws/TaliForth2/tree/master/tests

From the Tali Forth test code:

The windows version of py65mon reads directly from the console rather than stdin, and therefore doesn't work with pexpect on Windows (even though pexpect is now supported on Windows).

It's been a long time since I tried it on Windows. I only used it on Windows long enough to get it working for some users that were asking for it. As I recall, I think the issue here was that we needed a way to do non-blocking input (check if a character is available). If one of the Windows users comes up with a better way than what it does now, I'd be happy to merge a pull request.

I'm wondering if this is the best way to do it or if you have any suggestions, because we're probably not the first people to try something like this. Thanks!

I think the Tali Forth test suite is an outstanding use of Py65 and exactly the kind of thing that we should support. I think it's great that you are able to use Py65 for this but I also think that it has failed you in some way. The Tali Forth test runner is written in Python, and Py65 is also written in Python, but the test runner is shelling out to py65mon via pexpect. I think we should try to figure out some way for your Python code to call Py65 as a Python library. I'd like to know what things are preventing that. We could make some changes or add a new API to help.

[scotws]

@mnaberez Thank you for that -- for the record, what has been seriously impressive is how we've done some really mean things to py65mon, and it's proved again and again to be rock-solid.

@SamCoVT has taken point on this question and has a bunch of ideas, so I'm going to get out of his way here.

The one thing I had been wondering about is a way to freeze and load the complete machine state, something like a snapshot in VirtualBox -- not only save the memory, but also the registers, etc. That way, you could add a "freezepoint" -- like a breakpoint -- that saves the machine state to a given file when the machine reaches there, and then could do an extensive postmortem externally -- look at the stack, for instance.

The relevance for testing would be that you could start by loading a known state ("have"), run the simulator from a certain address onwards, freeze the state at some point, and then compare it automatically to a target state ("want").

That would, of course, require some easy to use format for the frozen state, but with the small sizes we're talking about here -- 64 Kib and change -- I wonder if you could get away with some compressed form of JSON.

[scotws]

@SamCoVT has re-written Tali's testing program to access py65mon directly -- see https://github.com/scotws/TaliForth2/blob/master/tests/talitest.py - so as far as I'm concerned, this issue is closed. The speed increase is amazing, now at less than two minutes, with no more dropped characters.

I cannot overstate how much being able to run a test suite in py65mon has improved Tali's code quality. Maybe it would be worth adding a few lines to the py65mon documentation on how to do this? I'd offer to do it, but I'm honestly still a bit hazy on what exactly is going on under the hood here.

[mnaberez]

@SamCoVT has re-written Tali's testing program to access py65mon directly -- see https://github.com/scotws/TaliForth2/blob/master/tests/talitest.py - so as far as I'm concerned, this issue is closed. The speed increase is amazing, now at less than two minutes, with no more dropped characters.

Nice!

I will close this issue, but I created a new one (#43) to track the request for snapshots.

I cannot overstate how much being able to run a test suite in py65mon has improved Tali's code quality. Maybe it would be worth adding a few lines to the py65mon documentation on how to do this?

I am so happy to hear that. I think automated testing is a great use case for Py65 and one that we should support and encourage. I agree we should have some better documentation about it. There's still an open request for testing (#39). Hopefully we can help with that one also and then do a write-up.

Thank you!