Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to clone smaller instances with junction points (Windows) or symbolic links (Unix) #4129

Merged
merged 3 commits into from
Aug 2, 2024

Conversation

HebaruSan
Copy link
Member

@HebaruSan HebaruSan commented Jul 25, 2024

Background

It's common for users to make extra copies of KSP1 so they can install different mod lists in each one. CKAN supports this by tracking multiple instances and providing a clone operation, which copies an existing instance to a new folder and adds it to CKAN.

Motivation

  • Each game instance contains about 5.5 GiB of files that come with the game and are necessary for it to run. Copying these takes a long time, and having multiple copies of them wastes a lot of space.
  • Each game instance also contains files generated during gameplay and while CKAN is operating that are relevant to that instance but not really to clones. For example, screenshots are relevant to the instance where they're created, but having multiple copies of them isn't generally desirable. Copying these into cloned instances also wastes some time and space.

Changes

  • Now there is an option to share the stock files between the clone and the original instance, either via ckan instance clone --share-stock or a new "Share stock files" checkbox in GUI:
    image
    which is done by making junction points (Windows) or symbolic links (Unix) for the following directories instead of copying them:
    • KSP_x64_Data (the main stock game contents folder)
    • KSP_Data (same but on Linux)
    • GameData/Squad
    • GameData/SquadExpansion
    • Some stuff from the now defunct launchers, just in case those are present
  • Any folders that are already symbolic links (e.g., if you're cloning a previous thin clone or if you're a mod developer with symbolic links from GameData to your source folders) will be duplicated in the cloned instance as symbolic links to the same targets
  • Now the following directories' contents aren't copied into cloned instances of KSP1; the directories are created, but left empty:
    • CKAN/history (old mod lists)
    • CKAN/downloads (deprecated but might still contain files in very old instances)
    • saves
    • Screenshots
    • thumbs
    • Missions
    • Logs
  • Equivalent updates are made for KSP2 instances

This will allow the creation of game instances that are much smaller but still playable (a thin clone of my Steam instance shrinks from 7 GiB to 81 MiB, or about 1% of its original size, and is very quick to generate).

Note that the CKAN registry and the currently installed mods will still be copied across, so users will still be able to set up some mods in one instance and then clone them elsewhere as a starting point for a different mod list.

Fixes #4087.

@HebaruSan HebaruSan added Enhancement New features or functionality GUI Issues affecting the interactive GUI Cmdline Issues affecting the command line Core (ckan.dll) Issues affecting the core part of CKAN Tests Issues affecting the internal tests labels Jul 25, 2024
@HebaruSan
Copy link
Member Author

HebaruSan commented Jul 25, 2024

Hey @ProgrammerFailure, this is the pull request for cloning with symlinks. You can click the Checks tab, then the Artifacts dropdown, to download a test build with the "Share stock files" checkbox in the cloning window:

image

I've only tested it on Linux so far, so it needs to go for a test drive on Windows before merge. If you have a chance, please try it out and let us know if you see any issues. Thanks!

@ProgrammerFailure
Copy link

Hey, this didn't actually create the symlinks for me. As expected, stock files weren't copied, but symlinks also weren't created. Tested with three preexisting CKAN instances, one mod-free, one dev install, and one heavily modded game. No symlinks for Squad were created.

I used the CKAN you provided, clicked share stock files, and cloned an instance to no effect. I also had to disable the windows batac warning.

@ProgrammerFailure
Copy link

Windows 11, if that helps

@HebaruSan

This comment was marked as resolved.

@HebaruSan HebaruSan added the In progress We're still working on this label Jul 25, 2024
@ProgrammerFailure
Copy link

No problem, still looking forward to it.

@HebaruSan

This comment was marked as resolved.

@ProgrammerFailure
Copy link

ProgrammerFailure commented Jul 25, 2024

I can confirm it works with asroot. CKAN could request admin or just make a bat file and tell the user to run it?

@HebaruSan

This comment was marked as resolved.

@ProgrammerFailure
Copy link

For what its worth, I'm fine with granting CKAN admin once per clone operation, and considering it saves 5-6 GB, I imagine others will be too.

@HebaruSan

This comment was marked as resolved.

@ProgrammerFailure
Copy link

ProgrammerFailure commented Jul 25, 2024

True. Plus, I suppose there are still people without admin access to their system? What do you mean by discoverability?

@JonnyOThan
Copy link
Contributor

CKAN actually tries very hard to make sure that it’s NOT running as admin. This causes countless problems with users who run elevated all the time. No one understands this stuff.

There’s gotta be a way you can request UAC elevation? I’m kinda surprised that CKAN wouldn’t have that permission level considering that it just created the directories, right? Maybe there’s some option for setting the permission when creating. Or can the clone be created elsewhere and then moved into position?

@JonnyOThan
Copy link
Contributor

Oh interesting, you have to start a new process: https://stackoverflow.com/a/573150

Maybe just run console ckan with all the parameters and —as root?

@HebaruSan

This comment was marked as resolved.

@ProgrammerFailure
Copy link

ProgrammerFailure commented Jul 25, 2024

No, I am not. Should I test with it enabled?

The article mentions developing with VS. I am doing this, but it's just a plugin, not an app.

@HebaruSan

This comment was marked as resolved.

@HebaruSan

This comment was marked as resolved.

@HebaruSan

This comment was marked as outdated.

@HebaruSan

This comment was marked as resolved.

@HebaruSan

This comment was marked as resolved.

@HebaruSan

This comment was marked as outdated.

@JonnyOThan
Copy link
Contributor

Just realized something: I never actually use symlinks on windows when setting up references; I use junctions (mklink /j). I've never had to switch to an admin account or elevate privileges. I'm not exactly sure how to do that programmatically but maybe we can just invoke mklink directly on windows?

https://stackoverflow.com/questions/3615129/how-to-create-a-junction-point-using-winapi

@HebaruSan
Copy link
Member Author

@JonnyOThan, thanks for the idea! That does indeed look like the better approach on Windows. We're only making symlinks to the absolute paths of directories, so the only difference for us would be the security stuff.

image

@HebaruSan

This comment was marked as resolved.

@JonnyOThan
Copy link
Contributor

I'm like 90% sure they can...I recall doing that at a previous job.

@techman83
Copy link
Member

@HebaruSan yes they do. It's how I used to point some games on a different drive when stream didn't support install locations. I've probably still got the PowerShell script for the task somewhere.

https://en.m.wikipedia.org/wiki/NTFS_links#:~:text=While%20NTFS%20junction%20points%20support,support%20for%20cross%2Dfilesystem%20links.

@HebaruSan

This comment was marked as resolved.

@HebaruSan
Copy link
Member Author

I think the problems I was having were caused by thinking that strings consist of 1-byte chars like in C, but they're actually 2-byte chars (the API requires byte counts to be specified in several places).
Latest push:

  • Junctions instead of symlinks on Windows
  • The checkbox is always visible and checked by default

Should be a lot more useful now that all that security craziness is gone. I'll probably merge this version unless I or anyone else finds an issue shortly.

@HebaruSan HebaruSan force-pushed the feature/symlinks branch 2 times, most recently from adcc369 to 2912a21 Compare August 1, 2024 02:49
@HebaruSan HebaruSan changed the title Option to clone smaller instances with symbolic links Option to clone smaller instances with junction points (Windows) or symbolic links (Unix) Aug 1, 2024
@HebaruSan HebaruSan removed the In progress We're still working on this label Aug 1, 2024
@HebaruSan HebaruSan merged commit a37eca5 into KSP-CKAN:master Aug 2, 2024
3 of 4 checks passed
@HebaruSan HebaruSan deleted the feature/symlinks branch August 2, 2024 04:45
@ProgrammerFailure
Copy link

@HebaruSan ,

I wanted to download this but it says that these artifacts have expired. Is there any way to access it without building CKAN myself?

@HebaruSan
Copy link
Member Author

@ProgrammerFailure yes, you can try the dev build here:
https://ksp-ckan.s3-us-west-2.amazonaws.com/ckan.exe

@ProgrammerFailure
Copy link

Thanks! Will that link always go to the current dev build?

@HebaruSan
Copy link
Member Author

Yup, it gets built and replaced every time we merge something to the master branch.

@ProgrammerFailure
Copy link

ProgrammerFailure commented Sep 10, 2024

On trying to make a symlinked instance, I get this error:

CKAN.InvalidKSPInstanceKraken: Exception of type 'CKAN.InvalidKSPInstanceKraken' was thrown.
   at CKAN.GameInstanceManager.SetCurrentInstance(String name)
   at CKAN.GUI.CloneGameInstanceDialog.<buttonOK_Click>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

And it doesn't make the instance. Did I mess up something? Reputation based protection is disabled.

@HebaruSan
Copy link
Member Author

HebaruSan commented Sep 10, 2024

Did I mess up something

Nope, an unrelated bug caused by a later change is causing the "copy from" instance's path to not fill in to this box:

image

I just pushed a fix for this, which our GitHub workflows will process and eventually generate a new dev build, which the previous dev build should prompt you to install when you launch once it's ready.

Thanks for catching that!

@ProgrammerFailure
Copy link

ProgrammerFailure commented Sep 11, 2024

I'm now getting this error:

CKAN.InvalidKSPInstanceKraken: Exception of type 'CKAN.InvalidKSPInstanceKraken' was thrown.
   at CKAN.GameInstanceManager.SetCurrentInstance(String name)
   at CKAN.GUI.CloneGameInstanceDialog.<buttonOK_Click>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

@ProgrammerFailure
Copy link

ProgrammerFailure commented Sep 11, 2024

Latest dev version, and the box does get autofilled
I turned off real-time protection and it didn't have any effect.

@HebaruSan
Copy link
Member Author

Thanks again, found and fixed another recent issue with that popup.
The next dev build you try should work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Cmdline Issues affecting the command line Core (ckan.dll) Issues affecting the core part of CKAN Enhancement New features or functionality GUI Issues affecting the interactive GUI Tests Issues affecting the internal tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature]: Symlinks for GameData
4 participants