How should the upcoming phet-io version be published?

[samreid]

From slack:

Sam Reid:house_with_garden: 12:26 PM
We have a short timeline for publishing a metacog-enabled PhET-iO wrapper for our client Stanford. We have been testing a dev version, but a lot of our tooling only supports production versions. We won’t have time to fully design and implement all of the PhET-iO functionality, say, to bring balancing act to a feature-complete 1.0. We are trying to decide between:
a) publish an “rc” and “production version” even though they don’t have full functionality or even full testing
b) modify our tooling so that we can test and deliver “dev” versions with some degree of testing

Jonathan Olson 12:27 PM
This is a wrapper you're talking about being deployed, or a simulation?

Michael Kauzmann:house_with_garden: 12:28 PM
I think this is a question for @Kathy. Another way to ask the question is what is the lesser of the following two evils?
We said that we never wanted to conduct a research study from a link from phet-dev (only ever from phet-io.colorado.edu).
We generally want all of the newly published phet-io sims to be at the same level of instrumentation for clarity and maintainability ease.
(edited)

Kathy Perkins 12:29 PM
was added to #dev-public by Michael Kauzmann.

Sam Reid:house_with_garden: 12:34 PM
@chrisklus and I discussed it and recommend that stanford use a dev link, so we don’t have a rushed RC or a confusing “1.0” balancing act phet-io release.

Chris Malley 12:34 PM
It would be more appropriate to test and deliver an rc version than a dev version.
12:35
RC at least implies some level of QA. dev does not.

Sam Reid:house_with_garden: 12:35 PM
So we can track the shas?

Chris Malley 12:36 PM
I don't understand the question. Anything created by PhET build tools has an associated set of shas in dependencies.json.

Sam Reid:house_with_garden: 12:37 PM
So true! I was thinking more of a release branch and ease of deploying updated versions. But we can always branch from a dev sha.

Chris Malley 12:37 PM
Have you considered a 'one off'?
12:38
Is this something that will need to be supported? Maintenance releases, etc.

Ariel Paul 12:39 PM
My vote would be a one off

Chris Malley 12:39 PM
Modifying process (or worse, build tools) to meet a deadline sounds like the wrong direction.

Ariel Paul 12:39 PM
And it could be something like "stanford-test-1"

Kathy Perkins 12:39 PM
a one-off seems appropriate here to me.

Michael Kauzmann:house_with_garden: 12:40 PM
Sounds really nice to me as well! Much easier to track and version than a dev, but it doesn't really fit with the RC-production flow.

Chris Malley 12:40 PM
And this question: Is this something that will need to be supported? Maintenance releases, etc.
12:41
... because one-offs (afaik) do not fit into the maintenance process. (edited)

Kathy Perkins 12:42 PM
Not maintained on any regular basis. If Stanford did another research study we would want to rebuild on current master, but its not going to need to be maintained like our normal sims.

Chris Malley 12:43 PM
@samreid said "We have been testing a dev version, but a lot of our tooling only supports production versions." Will tooling be a problem with a one-off? If so, which parts of "tooling"? (edited)

Ariel Paul 12:45 PM
My feeling would be this is done as a one off and ideally work continues at some point to bring balancing act to a fully instrumented and RC Ed version

Sam Reid:house_with_garden: 12:45 PM
I believe one-offs are published to phet-dev.colorado.edu instead of phet-io.colorado.edu, so we would still need to update our tooling. At the moment, this means a process for un-password-protecting some directories and not forgetting to re-password-protect them later

Ariel Paul 12:46 PM
Having an option to publish a one-off to phet-io seems reasonable doesn't it?
12:47
I feel like that wouldn't be a massive change (naively)
12:47
And this contingency will come up again

Michael Kauzmann:house_with_garden: 12:52 PM
I feel like that wouldn't be a massive change (naively)
Currently one-offs are treated as glorified dev deploys, so IMO after looking at code for 2 minutes, to do this "right" it may be a bit involved. The "not as right" solution that could be fast would be to optionally be able to deploy dev to phet-io.colorado.edu

Sam Reid:house_with_garden: 7:20 PM
I don’t feel like pushing one-offs or dev versions to phet-io website helps significantly. My favorite options are:
(a) create an actual RC and production version, even though it won’t be a “full” PhET-iO version.
(b) use a dev version from the dev server, and update the tooling accordingly (not difficult) (edited)
7:22
It seems the “one-off” process is meant to support a side-step from a side branch. But we want to move forward with master and continually improving dev versions.

Chris Malley 7:23 PM
I like (a). There's nothing that says a production version has to be fully tested. It can be RC tested as much as time allows, and published with known problems, and still be a production version. It's a schedule/quality trade-off.
7:24
A one-off or dev version IS going to have the same quality as a partially-tested production version. By making it a production version, we're being clear about the intent and trade-off. (edited)
new messages
7:25
It's still going through the RC process, and you publish it when it's good enough for what is intended.

Chris Malley 7:29 PM
I realize what I'm suggesting is a departure from PhET production-quality standards. But again, it's a trade-off. And I don't see the point in changing tooling to try to avoid the fact that we're going to "ship" something that doesn't quite meet the usual standards.

Michael Kauzmann:house_with_garden: 7:32 PM
I think that bringing one-offs through an RC process feels best to me because then we get everything that @pixelzoom is discussing without degrading the quality of the "general sim" on phet-io.colorado.edu. I worry greatly about having any variety of sim on phet-io.colorado.edu because we come back to sims and then want to repurpose them with short turnarounds. Cutting the feature implementation corner leads to more corners being cut in the future. I like the idea of consistency in the product that is on phet-io.colorado.edu: A "published phet-io sim" being feature complete at the very least (and as of last summer designed apis as well). (edited)

Chris Malley 7:33 PM
We don't have a process for bringing one-offs through RC. We have an RC > Production process.

Michael Kauzmann:house_with_garden: 7:33 PM
Well we can make a one off and give it to QA in an RC template. Boom process.

Sam Reid:house_with_garden: 7:33 PM
Michael, it seems your concern is varying quality on published sims on phet-io.colorado.edu, but we already have that, right?

Chris Malley 7:34 PM
Yeah.. What's the advantage to doing it as a one-off vs RC?

Michael Kauzmann:house_with_garden: 7:35 PM
Yes agreed sam, I'm trying to curb a current systemic problem and keep it from getting out of hand. Currently in our minds we have a good understanding of the instrumentation state of every published sim. This is in part because it is basically a duality between pre GQ and post with a few side notes. If we continue putting anything up there it will be harder and harder to know.

Chris Malley 7:35 PM
Whether it's called a dev, RC, or one-off, it's going to be the same product that gets shipped. If it's going to be shipped a little prematurely, I'd rather have it in an RC branch, where work (and maintenance) can continue on it if there's time/need.

Michael Kauzmann:house_with_garden: 7:36 PM
Can one offs get maintenance releases right now?

Chris Malley 7:36 PM
I don't believe so.
7:36
But I'll bet JO could make it happen ;)

Jonathan Olson 7:36 PM
only production versions really work for maintenance

Chris Malley 7:37 PM
so... more work/tooling to patch one offs.
7:38
It also seems to me that shipping something prematurely is a very strong argument for keeping it in the RC > Production process.
7:38
There's a higher probability that something will need to be fixed and re-released. (edited)

Michael Kauzmann:house_with_garden: 7:39 PM
There will be tooling changes either way. We will still need to figure out the list of actually completed PhET-iO sims. It just won't be the list of sims on phet-io.colorado.edu

Chris Malley 7:40 PM
How did this go from "a metacog-enabled PhET-iO wrapper" to "the list of actually completed PhET-iO sims"?
7:41
And why is identifying "the list of actually completed PhET-iO sims" not a production problem?

Michael Kauzmann:house_with_garden: 7:42 PM
Are you saying when a sim is in RC, that we don't do a production deploy on it, and deliver the RC on phet-dev?

Chris Malley 7:43 PM
No. Do RC until its ready to publish, where "ready" is in this case a schedule/quality trade-off.

Jonathan Olson 7:44 PM
So I'm a bit confused, what's the advantage to NOT doing the normal RC/production releases for the sims? We want to release them (even if it's limited and not yet complete), and we can track which ones are "complete"

Chris Malley 7:44 PM
Extra work.
7:45
And the word "one off" means what it says - something quick-and-dirty that's a one-time thing, not something we intended to maintain.
7:46
Sorry JO, misread your comment. My question is the same -- what's the advantage of not doing RC/production? (edited)

Michael Kauzmann:house_with_garden: 7:46 PM
That all sounds great. So for balancing act we will do an rc and then production phet-io deploy and then the study will use the phet-io.colorado.edu link, and will be automatically maintained.

Chris Malley 7:47 PM
If you're delivering it to a client, it's a production version. Calling it something else doesn't change the fact that you're delivering a lower quality 'product' as a conscious schedule/quality trade-off.
👍
1

7:48
And I think it's totally fine to do that. And like I said above, even more important to keep it in the RC/production process because it may have problems that need attention.

Michael Kauzmann:house_with_garden: 7:51 PM
Everything you are saying sounds great to me, but then the unspoken weirdness to me there is that it then sits nexts to a sim that is feature complete and ready for all uses. We need to come up with a way to differentiate.
7:51
Don't worry though, one issue coming in hot.

Chris Malley 7:52 PM
Document what's incomplete as GitHub issues, for that release branch.

Jonathan Olson 7:52 PM
I mean we have a DB for phet-io sims and their branches. Just add a "complete" flag?
👍:skin-tone-2:
1

Michael Kauzmann:house_with_garden: 7:53 PM
I mean we have a DB for phet-io sims and their branches. Just add a "complete" flag?
Yeah I'm thinking that too, as the perennial task that gathers a list of all published phet-io sims is already using the metadata.

Chris Malley 7:54 PM
I predict that if/when PhET-iO really takes off, this kind of thing (responding to specific clients) is going to become even more common. The more we can keep it inside the RC/production process, the easier it will be to manage.

Michael Kauzmann:house_with_garden: 7:55 PM
Right and the easier it is to know what the current status/feature completeness of a given sim is.
8:00
https://github.com/phetsims/phet-io/issues/1609

Ariel Paul 8:01 PM
The ability to do maintainance releases is a compelling argument vs my preference for a one-off earlier. I guess I always think of RC stuff as "taken to our full standards" but it does make sense to instead think of it as what it is "a release candidate"

Ariel Paul 8:02 PM
there is no expectation that RC.1 is actually ready to be published to the website

Michael Kauzmann:house_with_garden: 8:03 PM
But I think @pixelzoom and @samreid are recommending that these sorts of client specific sim pushes go through to a phet-io production deploy

Ariel Paul 8:03 PM
It is just the first version in the process
8:03
I mean, you are not sending that balancing act sim to other clients
8:03
So it does seem like a "production deploy" for the intent
👍:skin-tone-2:
1

8:04
It just needs to be clearly documented
8:07
I think the tricky piece will be a client specific on a short timeline when a well vetted version already exists
8:08
Balancing act hasn't been instrumented yet, but what if you need some specially tweaked iO version of say GQ to support a research study, we don't have to solve that right this second, but might want to consider the contingency

[zepumph]

To condense another ~20 minutes of slack conversation, it seems like we think that it is best to try to keep these "one off like" client specific sim requests in the RC/production pipeline as much as possible. This helps with maintainability, and ensuring that testing client facing product never diminishes.

The general consensus seemed to be that creating one-offs for these wasn't ideal. This is because it isn't obvious that a one-off would deserve the same testing attention as an RC (which we want because these sims are client facing), and also because one-offs are not built into the auto maintenance release process, which is important to maintain these client facing sims.

From this we may need to spend a bit of time thinking about how to differentiate a feature complete phet-io sim on phet-io.colorado.edu from something more like this balancing act release. See https://github.com/phetsims/phet-io/issues/1609 for more on that.

Main parties in the discussion were @samreid @pixelzoom @jonathanolson @ariel-phet @zepumph. Tagging so that all can comment if I didn't quite catch all nuance.

[samreid]

@kathy-phet asks: if we publish the phet-io brand without publishing the phet brand version, will that work ok? This won't disrupt the main site version at all?

[samreid]

We consulted with the dev team and the agreement was: it is safe to publish RC or production versions the phet-io brand without publishing the phet brand. We created an RC version here: phetsims/qa#473

Closing.