TMT enablement #83

jkonecny12 · 2020-08-08T15:24:39Z

Enable TMT for Simpleline unit tests and smoke test.

This will be used by Packit and could be run locally.

TODO:

Enable local execution with prepare step which has artifacts. (Right now it can't import simpleline).

thrix · 2020-08-08T15:28:46Z

plans/smoke.fmf

@@ -0,0 +1,9 @@
+summary: Run smoke test
+discover:


I believe discover steps does not need to be here. You are running a script in execute a script, thus discovered tests will be ignored I believe

@psss right?

Thanks for your review! Solved

thrix · 2020-08-08T15:47:01Z

/packit test

jkonecny12 · 2020-08-08T17:14:30Z

/packit test

poncovka

Looks good to me.

psss · 2020-08-11T09:38:18Z

If you set the unit test path to the root of the git repo, import simpleline will work:

summary: Run unit tests
description: Run all unit tests for the Simpleline project.
contact: Jiri Konecny <jkonecny@redhat.com>
duration: 10m
path: /
test: ./tests/units/run_test.sh

However it still fails on other deps:

tmt run -vvv plan --name basic
...
ModuleNotFoundError: No module named 'gi'

Adding required python module into the plan will fix this:

prepare:
    how: install
    package: python3-gobject-base

jkonecny12 · 2020-08-11T10:27:58Z

If you set the unit test path to the root of the git repo, import simpleline will work:
summary: Run unit tests
description: Run all unit tests for the Simpleline project.
contact: Jiri Konecny <jkonecny@redhat.com>
duration: 10m
path: /
test: ./tests/units/run_test.sh

Yes, I know but in that case the Packit will not test installed source but instead it will test always the source code. I want to create new plan which will be ignored by Packit and that will work for the local run. I just didn't had time to finish that yet.

Thanks a lot for the prepare info. You saved my time in debugging that ;).

jkonecny12 · 2020-08-11T12:36:22Z

UPDATED:

Split plans to be make a correct preparation and test execution for local and pr-tests.

jkonecny12 · 2020-08-12T07:57:57Z

@psss What do you think, any space for improvement of my new configuration?

jkonecny12 · 2020-08-12T07:59:29Z

@thrix any idea about when my TMT configuration may work on cruncher?

poncovka

I am sorry, but I don't understand it at all.

psss · 2020-08-12T08:40:34Z

Looks good. Perhaps, if you would prefer to have both tests defined in a single file, you can do it like this:

summary: Run unit tests
description: Run all unit tests for the Simpleline project.
contact: Jiri Konecny <jkonecny@redhat.com>
duration: 10m

/source:
    summary+: ' (source)'
    test: ./tests/units/run_test.sh
    path: /
    tag: local

/system:
    summary+: ' (system installed)'
    test: ./units/run_test.sh
    path: /tests
    tag: system

The extended summary is, of course, just an optional enhancement.

thrix · 2020-08-16T20:30:04Z

@thrix any idea about when my TMT configuration may work on cruncher?

Wanted to make last week, did not work out :/ .. so this week, I plan to work on it from tomorrow

jkonecny12 · 2020-08-17T09:45:09Z

UPDATED:

Updated based on @psss suggestion. Thanks a lot!

jkonecny12 · 2020-08-17T14:19:18Z

jenkins, test this please

TomasTomecek · 2020-08-17T14:50:18Z

/packit build

jkonecny12 · 2020-08-18T13:06:12Z

/packit build

thrix · 2020-08-19T06:49:12Z

Ah F33 has branched, I am fixing fedora-32 now.

thrix · 2020-08-19T06:49:24Z

/packit test

jkonecny12 · 2020-08-19T14:02:03Z

/packit test

thrix · 2020-08-19T17:27:13Z

Sorry I messed up, deploying another fix

thrix · 2020-08-19T17:29:27Z

/packit test

thrix · 2020-08-19T17:39:15Z

fixed :)

sakalosj · 2020-08-23T08:25:48Z

/packit test

sakalosj · 2020-08-23T08:45:25Z

/packit build

jkonecny12 · 2020-08-25T12:02:37Z

/packit test

jkonecny12 · 2020-08-25T12:06:58Z

/packit build

jkonecny12 · 2020-10-23T15:13:20Z

/packit build

poncovka

Looks good to me now. Thank you!

jkonecny12 · 2020-10-23T16:04:38Z

Seems this is still blocked by Cruncher:

 No 'script' defined in the execute step.

I hope it will be fixed soon :(.

thrix · 2020-10-26T07:44:14Z

@jkonecny12 we are almost there, but we got blocked on getting CentOS into AWS :/ Sorry for the delay

VladimirSlavik · 2020-10-26T13:39:20Z

Do we really need to have all these files all over the repository? Isn't there some way to have them contained in .fmf, or tests/tmt, or such?

Also, it makes no sense to me that we're enabling TMT but all related filesystem bits are called FMF.

psss · 2020-10-26T14:39:22Z

Do we really need to have all these files all over the repository? Isn't there some way to have them contained in .fmf, or tests/tmt, or such?

The .fmf directory marks the root of the metadata tree (similar to what .git does) and it should be in the git repository root. As far are tests and plans are concerned, it is possible to choose arbitrary location for them. For example like this:

tests/plans ... for L2 metadata
tests/tests ... for L1 metadata

But storing test code & metadata under tests and plan configuration under plans seem to be a reasonable default.

Also, it makes no sense to me that we're enabling TMT but all related filesystem bits are called FMF.

The fmf format is used to store metadata with support for inheritance and elasticity (you can look at it as an extended yaml) while tmt is the test metadata specification and tool for easily running them from your laptop:

Hope this helps.

jkonecny12 · 2020-10-26T14:47:35Z

@jkonecny12 we are almost there, but we got blocked on getting CentOS into AWS :/ Sorry for the delay

Looking forward to that! Thanks for the update.

VladimirSlavik · 2020-11-02T09:55:22Z

As far are tests and plans are concerned, it is possible to choose arbitrary location for them.

And that would happen in some configuration file? (Just asking to understand.)

But storing test code & metadata under tests and plan configuration under plans seem to be a reasonable default.

Well, both yes and no... I mean, from the testing point of view, certainly yes. But if I'm a wannabe contributor, open the repository and see the file listing, I see mostly stuff not really related to developing the thing. With some .hidden directories I can at least tell I can ignore them. Hope that explains why I'm asking.

Hope this helps.

Thank you, it does!

jkonecny12 · 2020-11-03T12:13:25Z

As far are tests and plans are concerned, it is possible to choose arbitrary location for them.

And that would happen in some configuration file? (Just asking to understand.)

But storing test code & metadata under tests and plan configuration under plans seem to be a reasonable default.

Well, both yes and no... I mean, from the testing point of view, certainly yes. But if I'm a wannabe contributor, open the repository and see the file listing, I see mostly stuff not really related to developing the thing. With some .hidden directories I can at least tell I can ignore them. Hope that explains why I'm asking.

Hope this helps.

Thank you, it does!

Hi @VladimirSlavik , please ask me directly. I think I'm able to answer most of your questions and it's much easier to discuss this on BJ/IRC than GH. ;)

martinpitt · 2020-11-10T16:23:52Z

plans/local/main.fmf

+    discover+:
+        filter: tag:local
+    prepare:
+        - name: Install missing dependencies


This syntax is invalid. I copied that in my own playground, and filed a wrong bug as I was misled by the failure output.

prepare: is a dict, not a list. See the documentation

Thanks a lot for your insight. Interesting, I'm pretty sure that worked locally :D

At the end this syntax seems to be supported, see:

https://github.com/psss/tmt/blob/master/plans/unit.fmf#L9

Seems that any stage could be specified like this to be able specify multiple different steps. However, it's not really required here.

Unfortunately, I can't find any sign about this in the documentation. Am I missing something @psss?

Yes, steps support multiple definitions. This is documented in the multiple configs section and included in the examples as well. Were would you suggest to add an additional mention about this?

@psss
Ohh, thanks a lot for the pointers. I totally missed the multiple configs section. I guess the problem is that it's just in the Examples section and not mentioned in the Specification part. So maybe, it would be nice to add this to Metadata Specification section too with explanation that both variants are valid.

martinpitt · 2020-11-11T05:54:14Z

FTR, I played around with that a bit, and https://github.com/martinpitt/python-simpleline/pull/1 now runs the tests correctly (except for a pylint version issue on older Fedoras, but that's not TMT's fault). Still requires some hacks, but these are said to go away with the TMT migration. So indeed let's wait with this for a bit until that happened. Until then, we have unit tests on github.

jkonecny12 · 2020-11-12T11:29:30Z

UPDATED:

Fixed bad syntax pointed by @martinpitt , thanks!

jkonecny12 · 2021-02-22T16:15:38Z

/packit build

jkonecny12 · 2021-02-23T11:54:04Z

/packit build

Removing my old nack -- LGTM if it works, thanks!

This configuration will work only for PR testing or gating.

PR-testing and Gating have to test unit-tests based on the installed package, however, when we are running these tests in local then we want to start the tests on source and not the installed package. Create two plans to solve this issue.

jkonecny12 added the enhancement label Aug 8, 2020

thrix reviewed Aug 8, 2020

View reviewed changes

jkonecny12 force-pushed the master-tmt-enablement branch 2 times, most recently from 96f2376 to 65fa2ad Compare August 8, 2020 15:58

poncovka approved these changes Aug 10, 2020

View reviewed changes

VladimirSlavik approved these changes Aug 11, 2020

View reviewed changes

jkonecny12 force-pushed the master-tmt-enablement branch from 9cbd640 to c7419f7 Compare August 11, 2020 12:54

poncovka reviewed Aug 12, 2020

View reviewed changes

jkonecny12 force-pushed the master-tmt-enablement branch from c7419f7 to 3479e5c Compare August 17, 2020 09:44

jkonecny12 force-pushed the master-tmt-enablement branch from 3479e5c to 77dee87 Compare September 23, 2020 11:43

poncovka approved these changes Oct 23, 2020

View reviewed changes

martinpitt previously requested changes Nov 10, 2020

View reviewed changes

jkonecny12 force-pushed the master-tmt-enablement branch from 77dee87 to 07bdd4f Compare November 12, 2020 11:29

martinpitt mentioned this pull request Nov 21, 2020

Enable copr builds for packit config cockpit-project/cockpit-ostree#31

Merged

jkonecny12 force-pushed the master-tmt-enablement branch from 07bdd4f to 12eeab5 Compare February 22, 2021 16:17

jkonecny12 added 4 commits February 23, 2021 13:56

Add TMT smoke test

7a61b1e

Add unit tests to TMT

36204c9

Move existing TMT plans under pr-testing directory

a3a416d

This configuration will work only for PR testing or gating.

jkonecny12 force-pushed the master-tmt-enablement branch from 12eeab5 to c8ef888 Compare February 23, 2021 12:56

jkonecny12 mentioned this pull request Feb 24, 2021

Builds hang in "Building SRPM" packit/packit-service#976

Closed

TMT enablement #83

Are you sure you want to change the base?

TMT enablement #83

Conversation

jkonecny12 commented Aug 8, 2020 • edited Loading

thrix Aug 8, 2020

Choose a reason for hiding this comment

thrix Aug 8, 2020

Choose a reason for hiding this comment

jkonecny12 Aug 8, 2020

Choose a reason for hiding this comment

thrix commented Aug 8, 2020

jkonecny12 commented Aug 8, 2020

poncovka left a comment

Choose a reason for hiding this comment

psss commented Aug 11, 2020

jkonecny12 commented Aug 11, 2020

jkonecny12 commented Aug 11, 2020

jkonecny12 commented Aug 12, 2020

jkonecny12 commented Aug 12, 2020

poncovka left a comment

Choose a reason for hiding this comment

psss commented Aug 12, 2020

thrix commented Aug 16, 2020

jkonecny12 commented Aug 17, 2020

jkonecny12 commented Aug 17, 2020

TomasTomecek commented Aug 17, 2020

jkonecny12 commented Aug 18, 2020

thrix commented Aug 19, 2020

thrix commented Aug 19, 2020

jkonecny12 commented Aug 19, 2020

thrix commented Aug 19, 2020

thrix commented Aug 19, 2020

thrix commented Aug 19, 2020

sakalosj commented Aug 23, 2020

sakalosj commented Aug 23, 2020

jkonecny12 commented Aug 25, 2020

jkonecny12 commented Aug 25, 2020

jkonecny12 commented Oct 23, 2020

poncovka left a comment

Choose a reason for hiding this comment

jkonecny12 commented Oct 23, 2020 • edited Loading

thrix commented Oct 26, 2020

VladimirSlavik commented Oct 26, 2020 • edited Loading

psss commented Oct 26, 2020

jkonecny12 commented Oct 26, 2020

VladimirSlavik commented Nov 2, 2020

jkonecny12 commented Nov 3, 2020

martinpitt Nov 10, 2020

Choose a reason for hiding this comment

jkonecny12 Nov 12, 2020

Choose a reason for hiding this comment

jkonecny12 Nov 16, 2020 • edited Loading

Choose a reason for hiding this comment

psss Nov 17, 2020

Choose a reason for hiding this comment

jkonecny12 Nov 19, 2020

Choose a reason for hiding this comment

martinpitt commented Nov 11, 2020

jkonecny12 commented Nov 12, 2020

jkonecny12 commented Feb 22, 2021

jkonecny12 commented Feb 23, 2021

jkonecny12 commented Aug 8, 2020 •

edited

Loading

jkonecny12 commented Oct 23, 2020 •

edited

Loading

VladimirSlavik commented Oct 26, 2020 •

edited

Loading

jkonecny12 Nov 16, 2020 •

edited

Loading