Suggestions and requirements for automating the test infrastructure

[vielmetti]

There is a whole set of test cases in https://github.com/htacg/tidy-html5/tree/master/test, and some scripts there to start to automate the test process. There could be more.

What I'm interested in is modernizing this test infrastructure. One possibility is to use something that uses TAP - see http://testanything.org/producers.html for a whole range of tooling - including tools that run in Windows.

TAP output looks like this:

1..4
 ok 1 - Input file opened
 not ok 2 - First line of the input valid
 ok 3 - Read the rest of the file
 not ok 4 - Summarized correctly # TODO Not written yet

A tool that I have used on a different project is bats at https://github.com/sstephenson/bats which is installable on Linux and OS X from within Travis and that runs as simple bash scripts that should be suitably easy to handle.

That all said, if anyone has strong feelings about automated testing, please speak up.

[geoffmcl]

@vielmetti, OOF, take care about the voluntary work done by others... over years...

Of course, always There could be more. ;=))

No idea about TAP! But would ask what it can do so much better that scripts?

About automated testing, do not see a need, but that is not to be taken as against it!

But as copiously explained several times before, some of which I added to the test/README.md, we do not really have unit tests for tidy... some of the current tests can only be verified visually... some test other specific things...

Somewhere else you wrote about a test that failed for you. When I looked at the reason for the test, it was about a crash/segfault because of Missing attr values... That is fixed.

You reported a small difference in output, using your version of tidy... not really part of the test... but when I run the tests in Windows 64-bit, and Ubuntu, using tidy 5.1.9, I have no problem.

So if your automated testing leads to what I think at this time is a false alert, then fix the automation before offering it... We would not want it to generate noise!

The reason I want to improve the test process is so that any developer that adds to tidy code has a way to quickly test their work before offering it as say a PR. Thus expect it to have been tested before merging to master...

If I think some developer has not done this, then I will usually try to find the extra time to do it for him/her... And, if it does cause a difference, explore it, discuss, decide if it is a valid change, and update testbase at the same time...

So at this time do not see how automated testing will help in this. If I am mistaken, then please enlighten this old fart ;-))

But what can you contribute to the current scripts?

They do need some TLC here and there... like another thing I read about a horrid cp tidy tidy5 in a yml script... Any script still using tidy5 should be fixed... not a new thing adjusted to perpetuate this 5... tidy is now tidy ;=))

Always open to what else can be done...

[vielmetti]

Just to be very clear -

I'm not saying that the current test infrastructure for win64, win32, Ubutnu 64 bit is broken. It seems to fit @geoffmcl 's needs very well. Take that as given.

What I'm missing is test infrastructure for Raspberry Pi 2: ARM7 (not Intel x86), running on 32 bits (not 64 bits), and Raspbian (not Ubuntu). It's a test of the whole system on a completely new chip, and thus it requires some care and scrutiny because none of those tests have been run by the lead developer or really by much of anyone. And thus any difference even tiny between the output of a test and its expected output might yield some insight about a portability problem.

Happy to do all of the test tooling myself in a fork, to make sure that I understand how all the pieces parts fit together. I do guarantee that it's going to generate some results where an Intel-only developer test will not find the same problems.

[geoffmcl]

Just to also be very clear -

What I hear is that you are only interested in what fits @vielmetti needs, and maybe finding some difference between Intel and ARM7 testing... no problem...

Agree we very much welcome extended porting to every system...

And when I am sure we are talking about the same code, meaning the same version of tidy, then will listen more carefully about differences in output when running the tests...

But more importantly how the code can be adjust to accommodate, fix the output change, and not break other major ports...

That could include some MACRO in the code like -

#ifdef SYSTEM_ARM
/* do this for @vielmetti */
#else /* for everyone else */
/* do that */
#endif

Hopefully minimal...

We would then also need some help from CMakeLists.txt configuration to generate say add_definitions(-DSYSTEM_ARM)... probably using like CMAKE_SYSTEM_PROCESSOR... Welcome research into this, and ideas...

And as read somewhere else, also some discussion, recommendations on package naming conventions, to include a CPU string... which would also start with something similar to build the full name string...

[vielmetti]

I have no intention of suggesting that the tests be peculiar to myself. The ARM platform is on millions of Raspbery Pi's, and I want to get that platform adequately tested so that the people who are packaging tidy-html5 for Debian and Raspbian can have some confidence that it's working.

i'm not ready to send in a PR yet, because too many of those have been rejected. You can look at

https://github.com/vielmetti/tidy-html5/blob/bats-firsttest/test/firsttest.bats

for the test rig I'm putting together with bats, and

https://travis-ci.org/vielmetti/tidy-html5/jobs/80416894

for a travis run that passed with a small set of tests. I'm purposefully skipping a set of test cases where the testbase output differs from the tidy output on Intel systems (Mac and Linux, clang and gcc) because those are all the same.

I had hoped that the Raspberry Pi (ARM) passes all of the tests that aren't skipped, but it doesn't, so investigating. I'll open an issue if I find something concrete.

[geoffmcl]

@vielmetti looks like you are having fun with travis-ci... Enjoy...

While suggestions for improving the test infrastructure are always welcome, not so much for automating, as already explained...

Anyway, before you can automate testing it seems a good idea to improve, understand, and discuss the testing...

We have already fixed some the old scripts, but there is more to do here first...

Solving some currently identified problems with just using the diff of two files as decision ctiteria... and not talking about just skipping it... but...

In the past I have had to exclude some original tests, and maybe there are more that are not relevant now... maybe some could come back... need to identify, discuss, and decide...

And probably some other improvements to the test infrastructure...

But the point is to not automatically run it as is, at this time, yet... but it's your travis-ci to do as you like...

As you state, please open an issue if you find something concrete. Of course if the something is related to current issues, then just add it appropriately... accordingly closing this.

As indicated, I prefer suggestions, requirements, ... be discussed on a list... build a consensus... get opinions... detail options, alternatives, and most importantly, benefit to tidy...

[vielmetti]

Thanks for making your intentions clear, @geoffmcl . I wish you luck in your project.