Add the remote plan with polarion tests

[jscotka]

Using remote tests for polarion testing: teemtee/tests#1

[psss]

The remote plan will not work until tmt-1.18 is enabled in Testing Farm. But the pull request already revealed an interesting thing: Thanks to inheritance the new plan has both plan and execute key, which should not be allowed. Let's discuss solution for this in #1648. As a workaround, we can disable inheritance for remote plans.

[psss]

Fresh tmt is available in Testing Farm, let's see what happens now! I've disabled the inheritance to prevent inheriting the execute step definition.

[psss]

Most probably failed because of tmt plan export tracebacking on remote plans: #1729

[psss]

We'll be able to verify this once tmt-1.20 with the fix for #1729 is deployed in the Testing Farm --> moving to 1.21.

[psss]

Everything should be ready in the tooling, so let's try if it works ;-) Rebased on the latest main as well.

[packit-as-a-service]

We were not able to find or create Copr project packit/teemtee-tmt-1646 specified in the config with the following error:

Cannot create a new Copr project (owner=packit project=teemtee-tmt-1646 chroots=['fedora-37-x86_64', 'fedora-36-x86_64', 'fedora-rawhide-x86_64', 'epel-8-x86_64', 'epel-9-x86_64']): Gateway Timeout
Consider using pagination for large queries.

---

Please check your configuration for:

1. typos in owner and project name (groups need to be prefixed with @)
2. whether the project name doesn't contain not allowed characters (only letters, digits, underscores, dashes and dots must be used)
3. whether the project itself exists (Packit creates projects only in its own namespace)
4. whether Packit is allowed to build in your Copr project
5. whether your Copr project/group is not private

[psss]

/packit test

[psss]

Tests are failing:

• /tests/core/dry because of non-existing directory
• /tests/discover/dynamic-ref/fmf because of an outdated fedora branch in @jscotka fork
• /requre/polarion because of recent logger changes in Decouple logging from objects and base classes #1685

@kkaarreell, do you think the dynamic ref test could be updated not to rely on the fork branches?

[kkaarreell]

@psss Does it? I would say it uses the branch of the project itself.

[kkaarreell]

We could hardcode tmt repo

[psss]

We could hardcode tmt repo

Sounds good.

[kkaarreell]

We could hardcode tmt repo

Sounds good.

maybe this would be it #1810

[adiosnb]

Tests are failing:

* `/tests/core/dry` because of non-existing directory

* `/tests/discover/dynamic-ref/fmf` because of an outdated `fedora` branch in @jscotka fork

* `/requre/polarion` because of recent `logger` changes in [Decouple logging from objects and base classes #1685](https://github.com/teemtee/tmt/pull/1685)

@kkaarreell, do you think the dynamic ref test could be updated not to rely on the fork branches?

I am trying to fix the test with --dry options with this #1811. What do you think?

[adiosnb]

@jscotka If tests in #1811 are OK, please merge jscotka#1 into this PR. Or we can discuss the next steps.

[psss]

@kkaarreell, @adiosnb, thank you both for the fixes. I suggest to merge #1811 to main when tests are green and rebase this pull request on it.

[psss]

Adding logger to the remote tests in teemtee/tests/pull/2.

[jscotka]

results of remote plan seems well,

[lukaszachy]

Polarion tests will need update after #1937 is merged, right?

[KwisatzHaderach]

Don't think so, there is one tests right now (tests/report/polarion), which is only in tmt repo and is disabled for obvious reasons.
What really needs to be done is moving the test to the tests repo and enabling it, definitely not worth doing as part of this patch.

[jscotka]

@psss I though that dry run issue is already fixed, seems not. reased and it still failed.

[psss]

Yes, one issue related to --dry has been fixed in 372a5d2. This looks like another one. @adiosnb, could you possibly have a look?

[adiosnb]

Yes, one issue related to --dry has been fixed in 372a5d2. This looks like another one. @adiosnb, could you possibly have a look?

Hi, I've changed the position of --dry option from the virtual step to global, and it fixed the test. The problem was that do my_run did not see the dry option set to True and tried to clone a remote plan into an uninitialized directory structure.

Old: tmt run --all --remove provision --how virtual --dry
New: tmt run --dry --all --remove provision --how virtual

@psss @jscotka What do you think about modifying the test in this way?

[adiosnb]

Yes, one issue related to --dry has been fixed in 372a5d2. This looks like another one. @adiosnb, could you possibly have a look?

Hi, I've changed the position of --dry option from the virtual step to global, and it fixed the test. The problem was that do my_run did not see the dry option set to True and tried to clone a remote plan into an uninitialized directory structure.

Old: tmt run --all --remove provision --how virtual --dry New: tmt run --dry --all --remove provision --how virtual

@psss @jscotka What do you think about modifying the test in this way?

I found another interesting behavior, which may lead to the actual bug.

Not working: tmt run --all --remove provision --how virtual --dry
Working: tmt run --all --remove provision --how virtual --dry plan

The problem with the not working case is that the dry option of Plan is set to True; thus, the remote plan is not cloned.

The question is, how is dry propagated to other objects?

[psss]

@psss @jscotka What do you think about modifying the test in this way?

I was looking at the test more closely and it seems it intentionally exercises the case when --dry is called under the provision step. So... I'm afraid we need to fix that bug. The global --dry is covered a couple lines above.

[psss]

Trying to fix the dry mode handling for remote plans in 0e0d10e.

[psss]

Thanks for preparing the remote polarion tests. I've fixed the problem with the provision --dry traceback for imported plans in 47b9250. Now all tests seem to be green.