Feature/cypress visual regression

[skjnldsv]

Closes #1339
1339 was becoming too messy 🙈

[cypress]

---

Test summary

24 • 0 • 0 • 0

---

Run details

Project	nextcloud-vue
Status	Passed
Commit	74fb30b ℹ️
Started	Aug 27, 2020 2:36 PM
Ended	Aug 27, 2020 2:37 PM
Duration	00:42 💡
OS	Linux Ubuntu - 18.04
Browser	Electron 83

View run in Cypress Dashboard ➡️

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

[skjnldsv]

I think we should find something later on to simplify this whole thing.
A bootsrap.js script to import that does all this?

[raimund-schluessler]

Ideally, we anyway should be independent of the server styles, I think. Would simplify visual testing and the styleguide build. At the moment we actually need to update the server styles with every release (although it was never done) and cannot be sure that it looks the same on every Nextcloud server version.

[skjnldsv]

Ideally, we anyway should be independent of the server styles, I think

I agree! But for fonts there isn't much choices :)
We could have our own base styling though, like variables.scss or icons.
That way we have a standard across our components to build their styles, that should help us and avoid having to write lots of unneeded small lines 😉

[raimund-schluessler]

The tests pass both on my local machine and on CI, so it is probably fine for the moment.

I am still a bit afraid that running the tests on different machines is a bit fragile and breaks if a developers setup is to different from the CI setup. But I guess we can adjust, if that happens.

[skjnldsv]

I am still a bit afraid that running the tests on different machines is a bit fragile and breaks if a developers setup is to different from the CI setup. But I guess we can adjust, if that happens.

The only issue was actually the fonts. We'll have to rely on this for now, but I'm pretty ok with this tbh :)

[ChristophWurst]

👍 🐘

[skjnldsv]

👍 🐘

@ChristophWurst I requested for reviews because I think this is something that should get our attention for existing and future components :)

You ok with our tests https://github.com/nextcloud/nextcloud-vue/blob/feature/cypress-visual-regression/tests/visual/components/AppSidebar/AppSidebar.visual.js ?

[ChristophWurst]

@ChristophWurst I requested for reviews because I think this is something that should get our attention for existing and future components :)

You ok with our tests https://github.com/nextcloud/nextcloud-vue/blob/feature/cypress-visual-regression/tests/visual/components/AppSidebar/AppSidebar.visual.js ?

I figured.

This is good stuff indeed. We just have to use it carefully. You don't want to test too much in this high level, these test tend to be very fragile and expensive (both in times to write, fix and maintain as well as how much time they need on CI).

For logic, covering as many paths as possible and the usual we should stick to isolated unit tests.

It's all about the balance :)

[skjnldsv]

It's all about the balance :)

Any ressources to determine the proper balance?
I guess we'll see in the long run when too much testing is too much?

[juliushaertl]

Very nice 👏

Great work @raimund-schluessler

[raimund-schluessler]

I think we don't need such tests for every component. But for components with many props and slots which all change the appearance and layout, we really need them. As #1281 and all the necessary follow-up PRs as #1288 and #1303 showed, AppSidebar is just to complex with to many different combinations of props and slots to modify anything without automatic visual tests. So this PR here should allow us to adjust such components with a lower risk of breaking anything.

[ChristophWurst]

Any ressources to determine the proper balance?

Usually the tests should looks like a pyramid where unit tests are the foundation and UI/pixel tests are the top. Which means you have lots of unit tests, some integration tests, a few visual tests and so on. The higher level it gets, the fewer tests you should have. But this is more of a guideline than a rule, as so many things :)