Proactively test v8 packages if they break on new Umbraco releases #63
-
|
Hey, I think it would be a huge benefit if HQ or package team could find an easy way to test v8 packages on affecting breaking changes with new Umbraco version. Maybe have a cloud site per active v8 package (there don't seem to be so many) and then roll out an upgrade and see which one breaks... You could even give the package dev access to the cloud site so they can test them selve... but upgrade can be handled by HQ/cloud... |
Beta Was this translation helpful? Give feedback.
Replies: 28 comments
-
|
This is a great idea - and something we have talked about before! Need to consider how it could be done the best way, should be automated somehow otherwise it would be a massive time-sink. Leaving it here with the idea label, if anyone has anything to add it would be appreciated! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks :) would just suggest to give every package dev a free cloud testing site, to put the packages on. HQ could then roll out the updates...and the package dev has an easy way to see if his package still works... |
Beta Was this translation helpful? Give feedback.
-
|
I don't see why giving package devs a free cloud site will help for testing new versions. I am pretty sure a package dev is able to spin up a new local installation of Umbraco for testing purposes. And for automated testing, how would you define whether or not Package A or Package B is compatible? It could seem compatible at first sight, but have some underlying issue somewhere. |
Beta Was this translation helpful? Give feedback.
-
|
Well for me it would help :) since it seems a lot easier to just have a site available that auto upgrades when a pre release is available... and you can just go in and test the package... instead of having to spin up sites for each release... compatibility issues are not a focus in this... just the dev person / team for his own packages.... |
Beta Was this translation helpful? Give feedback.
-
|
Like in case of the full text search (your package @skttl ) that breaks on v8.6 , that even YSODS a sites... so in that case cloud could send out a notification to you ... hey we upgraded but your package breaks.... |
Beta Was this translation helpful? Give feedback.
-
|
Yeah but most postupgrade errors doesn't surface as a YSOD. You are only going to catch a minimum of the potential problems, and instead you are going to get a false sense of security. Besides, with the amount of packages existing (even if we are envious of the amount of ie. Wordpress plugins), it would be a massive expense for HQ to run this kind of service. I'm all for some kind of automated testing, but I don't think this is the solution. |
Beta Was this translation helpful? Give feedback.
-
|
I still think it would remove frustrations and avoid releasing new umbraco versions that have to wait on package devs to discover the breaking changes and test them.... the responsibitly is still with the package dev to test the functionality... don't think there are an unhandleable amount of v8 packages (try to filter but doesn't seem to work https://our.umbraco.com/packages/?version=8), so expense and effort from HQ is doable I think... |
Beta Was this translation helpful? Give feedback.
-
|
IMO packages are just another dependency you choose to add to your project. And like any other dependency it might break if you do upgrades of the underlying platform or tech. It's up to us as devs to validate our dependencies on beforehand, wether they're Umbraco packages, nuget packages, stack overflow copy/pastes or whatever else. That being said .. maybe a system could be put in place where we as devs could flag packages as "currently not compatible with latest" towards the package devs, to give them an early heads up instead of them having to find out in their own. Then again, this system would probably just be the respective GitHub repos for the packages (at least the open sourced ones). |
Beta Was this translation helpful? Give feedback.
-
|
We had a discussion about this on today's package team meeting 🙂 As @skttl mentions, there is not a big difference in testing on Cloud vs testing locally. We discussed in the package team about releasing something like a preconfigured UI test solution with methods and examples for generating content, navigating to dashboards, etc. That pkg devs could then use to just configure the specific parts they need for testing their own packages. This test suite could then be run by the pkg devs themselves against new releases. |
Beta Was this translation helpful? Give feedback.
-
|
I think I first raised this with Kennith at the German festival 2019. At the time V8 was in...flux lets say...and kept breaking packages. I always like to look at these things from a couple of different angles, developer, business, marketing and what feels right but with a pragmatic head on. HQ have got cloud so you can see what packages people are running. Start with a list of the top 5 (or 2, or 10, or 20) packages. Focus on those and set up a handful of sites (a site can have multiple pacakages installed) that can run some UI testing on them over night and report back in the morning of any breaking changes. This is important as it brings "don't break packages" to the fore front of the HQ developers minds. If we do it the other way and push package devs to do all the work the night before a release (lets be honest we are all getting on with our lives) then its too late to make a change in core at that point so the fix will have to be in the package which means a package release (and the pressure that goes with it) and another thing to upgrade on the site. If on the other hand we can highlight the breaking change early then there might be time to rework the change or give a heads up to fix the package with plenty of time. This method need not be long term but it gives you a shield real quick. Currently when Umbraco breaks it doesn't look good in the eye of the client, which makes the agencies/developers look bad. It loses profit and good will through no real fault of anyone but good will can be eaten up real quick if it keeps happening. The rule of "always give it a few weeks before you upgrade Umbraco after a release" exists for a reason but it shouldn't, we should have faith that we can install it and forget (even if we aren't on cloud). To get there though HQ might have to do some more leg work than they would like to give us all more confidence and make it look like they are at least giving it a go to check popular package compatibility. Remember how we got here, people being encouraged to make packages with "this feature isn't on our road map, make it a package first" mixed in with a lack of contracts of what you could or couldn't touch in the back office. A hack here, a work around there forced by a missing extension point or a long lost PR. This is how you get in this state. But like I always said the fact we are talking about this stuff shows how far we've come, its a good think and not an excuse to bash HQ or Umbraco :) I love the idea of a test suite starter kit or something for package devs to create something with. Could be a nice idea to get others to contribute to packages other than the original author. Additionally maybe those packages with a test suite can get added to the automated over night build and get a new badge or promoted higher up the listings of packages. Something to promote/reward that extra level of commitment to quality. But if HQ aren't testing those suites against their code changes then its all pretty much for nothing as again HQ can break stuff but the pieces have to be picked up by devs in their free time trying to save their "online reputations" while balancing up family time, I don't think currently that is a good way of doing it. |
Beta Was this translation helpful? Give feedback.
-
The main problem there being we won't necessarily know the packages well enough to be able to know how to test them? Also how do we know what packages to test and which ones not to (uSync is a good example as it is a top package but not on Cloud)? And how does this scale? I am not against the idea at all, think it is great and the direction we should be moving towards - just need to figure out the right goal for this and what we can do sooner rather than later for this problem to be diminished 😊
This is a good point, I still don't think we can put the responsibility of writing the tests onto HQ or the package team. Only the package devs can be expected to know their package well enough to write them. In my mind the ideal solutions would be something along the lines of this:
This would not catch all errors ever, but would be a massive step up from the current situation while not putting unrealistic expectations against the package devs or HQ. |
Beta Was this translation helpful? Give feedback.
-
|
I do think HQ have the infrastructure available to make it easier on us "spare time" package devs (commercial ones have a good reason to make sure the packages work)... so why not share it? Happy devs, happy clients, happy HQ? And yes there is still the responsibility on the dev to do the testing/fixing but imaging that you could as an aspiring contributor see where a project needs help...could also attrackt new collaborators? Folks that are using the packages and immediately want to upgrade to new Umb versions... |
Beta Was this translation helpful? Give feedback.
-
|
@jmayntzhusen do you have an idea how many v8 open source packages there are? |
Beta Was this translation helpful? Give feedback.
-
|
@TimGeyssens on Our we currently have 145 packages listed for v8, that includes non-open source and is not a super reliable number but should give you and idea. And we have ~1300 published packages in total spanning Umbraco v4-8 (the bulk being for v7 though) |
Beta Was this translation helpful? Give feedback.
-
|
@jmayntzhusen thanks, so is it realistic to say that tops 20-25 devs would signup to an innitiative like this? |
Beta Was this translation helpful? Give feedback.
-
|
I think the first step is just to get something in place to act as a smoke test. As ever once something is in place it can be added to right? We know what the top most installed packages are from cloud (excluding usync) so could use that as a hit list. Jeffery from Perplex had a list of common packages from a few Code Gardens ago but a more up to date list should be able to come from HQ and cloud I would have thought. |
Beta Was this translation helpful? Give feedback.
-
That would be a good guess, but also just a guess - and wouldn't factor in potential growth at all 😉
Good point, I have some basic stats on downloads from Our, etc. But getting an overview of packages on Cloud should also be possible - will see what I can dig up 😊 |
Beta Was this translation helpful? Give feedback.
-
There seems to be a fear at HQ that once something is raised its binding, its not. All we ask is we discuss some of this stuff and try to find a solution that looks after as many as we can :) |
Beta Was this translation helpful? Give feedback.
-
Yeah good stuff with the package team, for me this would also be one that would tick the bill lowering barriers of entry and getting more people involved and interested in making packages |
Beta Was this translation helpful? Give feedback.
-
|
And I would be a bit demotived if the package team does all this effort to make it easier to start with packages but then the core team doesn't even care about breaking changes in these packages... |
Beta Was this translation helpful? Give feedback.
-
|
@jmayntzhusen maybe add a link to this discussion to the email that is going out? So we can hear some more opinions from package devs |
Beta Was this translation helpful? Give feedback.
-
|
Yep this will be included in the newsletter coming out next week as it is a very good discussion 🙂 Just want to clarify something: That being said I agree that more testing, more notice of changes and more tools in the hands of the package devs are a good thing, and definitely something we want to look further into! I am excited to see where this discussion may lead 😊 |
Beta Was this translation helpful? Give feedback.
-
|
Euhm don't think you are correct, it failed when upgrading the umb version to 8.6 so it is definitly a breaking change in the core that was the root issue... so yeah I would add it to the list... a community member already fixed it in the package, so you can view the details in it's sourcecode... Do you have an overview of the packages that are being tested by core? And how can we add more to that list? Cheers, |
Beta Was this translation helpful? Give feedback.
-
|
yeah to be fair to @jmayntzhusen - the failure in the FullTextSearch package was because of a missing attribute on the composer, meaning it tried to run while the Umbraco site was in Upgrade mode (see skttl/umbraco-fulltextsearch8@a7e07ed) - this means it would have been an issue during any v8 upgrade. I don't think this detracts from the main points of testing/checking. I too have been victim to the breaking change issues in the past, but its worth clarifying it hasn't actually been an issue for most of the recent releases. (not for me anyway 😉 ) |
Beta Was this translation helpful? Give feedback.
-
|
ok thanks for the input @KevinJump but I do think this should be handled in core... since any package that doesn't implement the attribute.... |
Beta Was this translation helpful? Give feedback.
-
|
yeah it sort of is :( The documentation isn't that clear on it (and this suggests that might need an update), but if your composer is a IUserComposer then it only runs when the RuntimeLevel is Run. https://our.umbraco.com/documentation/implementation/composing/#types-of-composers |
Beta Was this translation helpful? Give feedback.
-
|
ahh its here ... https://our.umbraco.com/documentation/implementation/composing/#runtimelevel I suspect though that documentation is much newer than the package 😞 |
Beta Was this translation helpful? Give feedback.
-
|
Umbraco now uses a stricter versioning approach, so breaking changes should only happen in majors and are included in the release notes and for bigger changes announced in advance. The CMS now uses Playwright to do end-to-end testing and installs/runs a new Umbraco instance on the build server, maybe that can be used as inspiration to create something similar to test your package on multiple different Umbraco versions? This requires writing the tests for your package first and from what I can see, most packages don't have those to begin with. |
Beta Was this translation helpful? Give feedback.
Umbraco now uses a stricter versioning approach, so breaking changes should only happen in majors and are included in the release notes and for bigger changes announced in advance.
The CMS now uses Playwright to do end-to-end testing and installs/runs a new Umbraco instance on the build server, maybe that can be used as inspiration to create something similar to test your package on multiple different Umbraco versions? This requires writing the tests for your package first and from what I can see, most packages don't have those to begin with.