Find untagged tests that need adjusting for OCIS

[PVince81]

Find all OCIS related tests where the behavior is not aligned with OC 10 but where alignment is desirable.
Such tests exist only to enforce the current non-compliant behavior and make sure we can detect changes in such behavior in case of regressions or fixes.

Currently we've been looking at @skipOnOcis tags followed by issue tags to find what API tests still need to be enabled.
However the above test category don't always have such tags, and are usually tagged as @skipOnV10.

I'm not sure if @skipOnV10 is enough to find those tests. We might want to add an additional tag that signifies its divergence from the expected behavior, and not just an accepted divergence.

Then raise tickets for all these cases if none exist already. Then add the issue tag there.

From owncloud/core#37624

@phil-davis @individual-it

[phil-davis]

@individual-it and I discussed and we think that the following tag combinations will allow people to easily understand why a scenario is skipped on OCIS or OcV10, and allow quick grep through the feature files to find counts of scenarios that are in each of the states:

• if a feature is working the same on oC10 and OCIS then there are no special tags.
• if a feature is implemented differently (better) on OCIS than oC10, and the divergence has been decided to be acceptable/good then there will be 2 scenarios. Tag them both with [skipOnOcV10 / skipOnOCIS] and divergenceIsAccepted
• if a feature is never to be implemented on OCIS, then tag it skipOnOCIS and notToImplementOnOCIS
• if a feature is not implemented at all yet on OCIS, then tag it skipOnOCIS and toImplementOnOCIS
• if a feature is implemented on OCIS but not working properly yet, then tag it skipOnOCIS and toFixOnOCIS, and tag the bug-demonstrating scenario for OCIS as skipOnOcV10 and toFixOnOCIS (or toFixOnOcV10 if the problem is actually in oC10)
• if a feature is never to be implemented on OcV10, then tag it skipOnOcV10 and notToImplementOnOcV10 (this might happen if there is some new thing that we do in OCIS and decide not to "backport" into OcV10)

[PVince81]

sounds great!

[PVince81]

For now, the obvious ones we can tag:

• old public webdav can already be tagged as notToImplementOnOCIS
• check [Tests-Only] Adjust updateShare.feature:179 for reva changes core#37624

For later, we can go through all @skipOnOcis, but only after we have less tickets to fix.

[PVince81]

also apply the same idea to the Phoenix repo (see owncloud/web#3737)

[PVince81]

now wondering if that's really a good idea... thinking of cases where EOS and OC storages behave differently, but in an expected way... so the matrix of possible tags would blow up somehow like "toImplementOnEOS" and "toImplementOnOC" unless we agreed to always only keep a general "toImplementOnOcis" where we mean any of the storages

[phil-davis]

toImplementOnOCIS should be fine - whenever we then try to implement something, I hope we implement it for all storage backends!

And if something is not implemented or not working on some particular backend then it will have one of the tags like skipOnOcis-EOS-Storage plus toFixOnOCIS - and that is obvious which OCIS part needs fixing.

[PVince81]

right, makes sense 👍

[phil-davis]

PR owncloud/core#37717 provides the ability to define a file containing a list of expected failures. That will let us manage the list of failing scenarios locally in each repo that uses the core API acceptance tests as their "standard end-to-end tests" for checking compatibility with oC10 WebDAV, OCS etc APIs.

That will reduce the need for much of the tagging in core, and so will reduce the effort of coordinate all the taging between core and the other repos that use the test scenarios.

[phil-davis]

We ended up doing:

• tag toImplementOnOcis for core API scenarios that are for features that are not at all near implemented on OCIS yet, but [should|might] be implemented one day. Those scenarios are not run in normal OCIS CI because they would be a complete waste of CI time, and they might crash the CI in some "bad" way (e.g. because the API end-point does not exist at all and so the response to the test code is some completely [weird|unexepcted] thing) Over time, as work starts on implementing these features, we will remove this tag, and the scenarios will start running in OCIS CI.

• tag 'notToImplementOnOcis` for core API scenarios that test features that are not intended to be implemented on OCIS. These are not run in OCIS CI, and we do not ever expect to run them.

• other API test scenarios get run in OCIS CI. The ones that currently are known to fail are listed in an expected-failures file in the relevant OCIS repos (currently owncloud/ocis owncloud/ocis-reva and cs3org/reva )

• OCIS bug demonstration scenarios have been moved into the OCIS repos and are run in a "local API acceptance tests" CI pipeline. So there is no longer any need for skipOnOcV10 or toFixOnOcis tags in core.