Add RO/RP-1 express install netkans

[NathanKell]

Purpose

This adds "express install" packages (a main package and three choices for graphics level) to allow express installation of RO/RP-1/RSS/etc.

Motivations

RO/RP-1 users often ask for a simple gamedata zip, or "what mods should I install", or the like--to the point of asking which recommended or suggested mods to install, which graphics mods, etc.

Changes

This adds a single express install netkan that depends on the core mods for an RP-1 install, and then depends on an RSSGraphics package. That package is provided by three express packages, each reporting only as a detail level, which depend the various RSS-Textures packages, visual enhancements, and Canaveral terrain/pads necessary for a complete experience.

Metanetkans for easy reference:

• https://raw.githubusercontent.com/KSP-RO/RP-1-ExpressInstall/main/RP-1-ExpressInstall.netkan
• https://raw.githubusercontent.com/KSP-RO/RP-1-ExpressInstall-Graphics/main/RP-1-ExpressInstall-Graphics-High.netkan
• https://raw.githubusercontent.com/KSP-RO/RP-1-ExpressInstall-Graphics/main/RP-1-ExpressInstall-Graphics-Medium.netkan
• https://raw.githubusercontent.com/KSP-RO/RP-1-ExpressInstall-Graphics/main/RP-1-ExpressInstall-Graphics-Low.netkan

Issue ref: #8688

Once KSP-CKAN/CKAN#3426 is merged, I will update the local netkans with comments on the graphics choice.

[NathanKell]

Side note: I just learned NetKAN's inflation error reports the line number and error for an internal JSON representation of the YAML spec. I was at first quite surprised to hear of an error on a line, and with a character, that my file did not include!

[NathanKell]

I believe this is now correct, though the checks are still failing--this seems to be because the CLI version of CKAN defaults to installing recommended mods, and the main netkan depends on both Ven's New Parts (which recs Ven's Stock Revamp) and on ReStock, and of course VSR and ReStock conflict.

Is there some way to get the checker to run without auto-installing recs?

[HebaruSan]

Side note: I just learned NetKAN's inflation error reports the line number and error for an internal JSON representation of the YAML spec. I was at first quite surprised to hear of an error on a line, and with a character, that my file did not include!

Does it? I don't think that's the case.

[HebaruSan]

I believe this is now correct, though the checks are still failing--this seems to be because the CLI version of CKAN defaults to installing recommended mods, and the main netkan depends on both Ven's New Parts (which recs Ven's Stock Revamp) and on ReStock, and of course VSR and ReStock conflict.

Is there some way to get the checker to run without auto-installing recs?

The checker is highlighting a real problem. These probably shouldn't recommend (depend on? not sure yet) conflicting modules.

[HebaruSan]

Oops, forgot these were metanetkans, I'll remove those depends, but can you add them to your repo?

[HebaruSan]

We'll need to copy this metadata locally and try installing with GUI to make sure the desired work flow happens. The command line tools are already easier for the RP-0 identifier because they don't interactively prompt for recommends or suggests.

[HebaruSan]

We'll need to copy this metadata locally and try installing with GUI to make sure the desired work flow happens. The command line tools are already easier for the RP-0 identifier because they don't interactively prompt for recommends or suggests.

What I'm trying in case anyone wants to replicate it:

1. Make a CKAN-meta-rp1express folder somewhere locally (doesn't matter where)
2. Create RP-1-ExpressInstall-v1.0.ckan, RSSGraphics-ExpressInstall-Low-v1.1.ckan, RSSGraphics-ExpressInstall-High-v1.1.ckan, and RSSGraphics-ExpressInstall-Medium-v1.1.ckan in that folder with their contents as reported by the validation script
3. Make an archive of that folder: tar czf rp1express.tar.gz CKAN-meta-rp1express
4. Add the archive as a CKAN repo: ckan repo add rp1express 'file:///full/path/to/rp1express.tar.gz'
5. Launch GUI, click Refresh to load the new repo, start testing
   

[HebaruSan]

My compatible versions are 1.10–1.12. Initial change set:

Click Apply, one recommendation appears:

Click Continue, prompted to choose RSS textures, I'll install the 2K package:

Prompted to choose KSPBurst or KSPBurst Lite, I'll pick the non-lite one:

Prompted to choose RSS Graphics, I'll try High to see how it handles a mismatch with the textures:

Install fails:

So it doesn't look like the RSSGraphics-ExpressInstall relationships are working as intended. I was under the impression that that choice was supposed to be presented early and drive most of the rest of it. Probably need to reorder the depends list a bit.

[HebaruSan]

You don't need to use any_of for MechJeb because the dev builds have "provides": "MechJeb2". A simple depends on MechJeb2 will prompt the user to choose if they have the dev repo configured. The any_of just makes the dev build show up twice:

[DasSkelett]

So it doesn't look like the RSSGraphics-ExpressInstall relationships are working as intended. I was under the impression that that choice was supposed to be presented early and drive most of the rest of it. Probably need to reorder the depends list a bit.

Even reordered it prompts me for a RSSTextures resolution later on. We might have to fix our RelationshipResolver behavior to recalculate it after TooManyModsProvides selections have been made.
Although thinking about it, doesn't it basically restart the installation from start after a selection has been made? So it should already include them for relationship resolution, no? 🤔

[HebaruSan]

Yeah, this is where it should skip that relationship:

https://github.com/KSP-CKAN/CKAN/blob/a8331592dda3bbfed3e42cdeb5e39f6858d23c5c/Core/Relationships/RelationshipResolver.cs#L381-L385

And this is where that call should already be checking provides:

https://github.com/KSP-CKAN/CKAN/blob/a8331592dda3bbfed3e42cdeb5e39f6858d23c5c/Core/Types/RelationshipDescriptor.cs#L113-L118

So... deeper investigation required.

[HebaruSan]

I wonder if the ordering of the depends inside the low/med/high modules is affecting it. Depending on RSS before the textures might make it pull in RSSTextures before the specific texture dependency is applied:

      "depends": [
          {
              "name": "RealSolarSystem"
          },
          {
              "name": "RSSTextures4096"
          }
      ],

EDIT: Nope, fixing that (putting RSSTextures4096 first) still doesn't suppress the RSSTextures prompt. 😕

[NathanKell]

I believe this is now correct, though the checks are still failing--this seems to be because the CLI version of CKAN defaults to installing recommended mods, and the main netkan depends on both Ven's New Parts (which recs Ven's Stock Revamp) and on ReStock, and of course VSR and ReStock conflict.
Is there some way to get the checker to run without auto-installing recs?

The checker is highlighting a real problem. These probably shouldn't recommend (depend on? not sure yet) conflicting modules.

Perhaps the later bit (where you're attempting to install via GUI) supersedes this but, see here for RP-0's netkan. The issue arises because the express install netkan depends on Restock, but in RP-0.netkan Ven's is the first of the two any_of recs so it is preferentially installed as the CLI blasts through every single mod's recommendeds.

[HebaruSan]

Ven's is the first of the two any_of recs so it is preferentially installed

I'm looking for code that could do this, and I'm not finding it. The resolution of the any_of rec should return both modules, which should then cause the user to be prompted to choose.

[NathanKell]

@HebaruSan ah then I misunderstood, so the issue is actually worse in CLI than I thought, i.e. it will turn "any_of" into "all_of"? :D

[HebaruSan]

@HebaruSan ah then I misunderstood, so the issue is actually worse in CLI than I thought, i.e. it will turn "any_of" into "all_of"? :D

No, it should be prompting.

[NathanKell]

Oh, even prompting in CLI in the action? Yeah that does seem weird.

Well, I'm happysad I provided a great way to find client bugs, I guess? :\

[NathanKell]

( MM dependency added here KSP-RO/RP-1-ExpressInstall-Graphics@3121b77 per note upthread.)

[HebaruSan]

EDIT: Nope, fixing that (putting RSSTextures4096 first) still doesn't suppress the RSSTextures prompt.

I think we're getting tripped up by the relationship resolver working depth-first, essentially. After we pick RSSGraphics-Whatever out of the list, we restart the install with two modules in the change set, RP-1-ExpressInstall and RSSGraphics-Whatever, in that order. RSSGraphics-Whatever is able to satisfy relationships itself (this is why we don't immediately get prompted to make the same choice again), but its dependencies are not looked at yet; rather, the relationship resolver tries to satisfy all of RP-1-ExpressInstall's dependencies (and depends-of-depends) before it looks at any of RSSGraphics-Whatever's depends.

So the current RP-1-ExpressInstall module approach does not seem to be workable. Rather, it seems that the module the user chooses up front needs to have all the "deep depends" in it, so the new modules could be something like:

• RP-1-ExpressInstall-Low
• RP-1-ExpressInstall-Medium
• RP-1-ExpressInstall-High

Each of those modules would have the depends that are currently in the RSSGraphics modules (with the detail-specific depends such as the 4K textures as early as possible in the list), and then the depends that are currently in RP-1-ExpressInstall, later in the list. That way it should be possible to pre-satisfy the RSSTextures and other dependencies early on.

Sorry for all the confusion, but this is getting really deep into the functionality of the relationship resolver, and we usually treat it as a black box not to be touched unless we're really sure what we're doing.

[HebaruSan]

Alternately, if you really want to have a "main" module branching into "detail level" modules, the "main" module could depend only on those "detail level" modules and not on RP-0, RO, RSS, etc.

[NathanKell]

Ah, I see what you mean. And you answered the question I was in the middle of asking. It is definitely the case that I want a main module, because as I understand the spec, I need to include an installable with each package or it gets treated as a metapackage and (per an issue from lamont a few years back) conflicts won't work.

I'll make that change.

[NathanKell]

Also, that route is required as well to take advantage of KSP-CKAN/CKAN#3426 - we need a main package to have a depends with a comment.

[NathanKell]

@HebaruSan this change is now done. 🤞

[HebaruSan]

OK, I'm taking a look now and I restarted the validation script. We might still need to reorder some of the depends, but the overall structure is a better fit to how the relationship resolver works now.

[NathanKell]

Thanks! And thanks for taking the time to dig through this all (especially the deep rabbit holes of CKAN black magic :D )

[HebaruSan]

Next test. Initial change set is a lot smaller, as expected:

Click Apply. One recommendation again:

Then the detail level prompt comes up:

I'll pick Low. Womp womp, the next prompt is the textures again:

Try putting the 4K/8K/whatever depends as early as possible in the list, before anything that could possibly depend on anything that depends on RSSTextures.

[NathanKell]

@HebaruSan Done, and inflator restarted.

[HebaruSan]

I need to take off for an event in a bit, but I'll be back in a few hours. The steps I listed above should still work if you're feeling adventurous.

[HebaruSan]

@DasSkelett, if ckan_meta_tester published its repo tar.gz file as a downloadable artifact, that could make it a lot easier to do client testing like this.

[NathanKell]

WOOHOO! (although sad robot noises at the beleaguered actionrunner :D )

[NathanKell]

Per my mistaken PR, in trying the express installer out I still got the prompt to choose KSPBurst version. This is despite KSPBurst-Full being the second line of the depends, here: https://github.com/KSP-RO/RP-1-ExpressInstall-Graphics/blob/main/RP-1-ExpressInstall-Graphics-High.netkan#L15

[HebaruSan]

I think that means you've got some older versions of the metadata active, from before the depends was updated to -Full. Make sure to remove any testing repos and click Refresh to get the latest official metadata.

[NathanKell]

Nope, that was the first thing I did on launching CKAN. :(

[NathanKell]

(To double check I closed CKAN, started it again, and refreshed again, same flow.)

[HebaruSan]

Check your repositories.

[NathanKell]

Only one listed, default, pointing to the official archive.

[NathanKell]

[HebaruSan]

OK fine, but it still isn't possible to pull in KSPBurst-Lite by depending on KSPBurst-Full. Some other weird thing is happening that you're not hinting at. Is anyone else able to reproduce this?

[NathanKell]

It is not possible to pull it in from depending on KSPBurst-Full, but RealAntennas depends on just KSPBurst, and we depend on RealAntennas. Which I think would make it the same sort of issue as the RSS-Textures thing?

I asked on the RO discord for anyone available to try installing in a blank KSP 1.10 install, hope I'm just cursed. :D

[NathanKell]

Ugh, and same with the sunflare. Yep, looks to be exactly the same dependency-resolving issue as before with RSS-Textures, and ordering in the netkan has no effect.

[HebaruSan]

It is not possible to pull it in from depending on KSPBurst-Full, but RealAntennas depends on just KSPBurst, and we depend on RealAntennas. Which I think would make it the same sort of issue as the RSS-Textures thing?

If that was the case, it would happen when I try it.

[NathanKell]

I have no explanation for why I am enountering that and you are not, unless I somehow have cached metadata that removing the repo it is from did not fix. Gonna clear my CKAN cache.

[HebaruSan]

You really were trying all of this in parallel while we were looking at it, though? At which point did this prompt re-appear for you?

[NathanKell]

Ah, I think we had a miscommunication then maybe? I had the prompt for KSPBurst, you suggest depending on KSPBurst-Full and moving it above. I did so and you responded that it fixed the GUI here ( #8701 (comment) ) , so I did not do a local retest at that point. Once it was merged down and in the main CKAN repo I tested now and got the prompts.

[NathanKell]

Ok, this just got weirder. DRVeyl reports:
"I was presented with EER and one other as recommended mods... then the textures [which we all expect]... then the TF configs. Then it started." With this image:

[HebaruSan]

Yeah, that one I saw. I assumed you changed something after my last test.

[NathanKell]

I do not get that prompt.

Wait, what KSP version were you testing against, and what compat versions? I bet that's it. I am on 1.10 with no compat versions; DRVeyl is on 1.10 but with 1.8 and 1.9 checked.

[HebaruSan]

Yup, TestFlight got moved around in the High Low metanetkan:

[NathanKell]

Ugh, that would do it. Thought I discarded that. :(

[HebaruSan]

I do not get that prompt.

That's even more evidence that you're not testing with the current metadata. Maybe try a completely fresh game install?

[NathanKell]

Will do.

[NathanKell]

Ok, a completely fresh install and linking that to CKAN fixed the issue, so now I'm only seeing evidence of my other stupidity 😂 (the TestFlight thing) rather than that weird situation with busted metadata. I fixed the local netkans so that should go away as well.

[NathanKell]

Circling back to say--it's working! Thank you again, from all of us--besides being I think a very positive option for our users, it'll make supporting them way easier on us. CHeers!

[HebaruSan]

That's awesome and great to hear! 🎉
Where are you expecting to see the bulk of your support feedback? The KSP-RO Discord maybe? I can stop by to observe now and then to see how it's going rather than making you serve as a go-between for everything.

[NathanKell]

Yeah, on the RO discord and sometimes on reddit. The forums are pretty quiet these days. But the issues are often less specific questions from installation (though as I've mentioned we do get the "which of these recommended mods should I really install" :D ), and more we find out a good few messages in that it's actually a broken install in one way or another, not a bug in (whatever submod).

And that's super kind of you! You should know that (AFAIK) next to none of the support load is actually from how to use CKAN--your docs, and our CKAN-101, seems to work pretty well, the only issue there has been walking people through adding the MJ-dev repo (which is why I've been poking at sarbs and lamont to merge dev to release, which they did so yay!). Well, that and the "ok but what should I really pick", so having a one-stop "just click this one thing" answer will be super awesome.

[HebaruSan]

I always forget KSP-CKAN/CKAN#2682 when repository shenanigans happen. 🙄

I think your CKAN client thought your metadata was up to date, because the Etag was the same as the last time you updated, and so it didn't actually do a refresh. Meanwhile you had removed a testing repo but its metadata was still in your registry, and would remain until the master repo's Etag changed. We don't have a good workaround for this currently; I sometimes edit my <KSP>/CKAN/registry.json file and delete the last_server_etag property to force a refresh.

[NathanKell]

AHA! Yes that makes perfect sense. I'll keep that in mind in future, it perfectly explains what was going on. The fresh install (or nuking the local CKAN folder, which presumably would have fixed it too but I wasn't taking chances, fresh install it was!) sorts that out by dint of utterly annihilating the json rather than tweaking it. :]