Folders per Version

[valc93]

I had some issues with updating nodejs and ruby. Both, when updated to the newest version, lost all ruby gems and npm packages. I had to re-install them all. I searched through scoop's file directory and found that my packages are still there but in a folder with the previous version number. Is it possible to make nodejs use a single folder with all its packages and not create a folder per version?

[ntwb]

I have been messing about with this the past month...

I am leaning towards having a npm config file in $HOME/.npmrc and setting a prefix folder to tell npm where to install the 'global' npm packages.

An 'out of the box' install of NodeJS from nodejs.org defaults to saving npm's global packages in \AppData\Roaming\npm (and cache in \AppData\Roaming\npm-cache) though setting this as the npm prefix I had issues with Scoop's npm shims.

At the moment I am using $HOME\\appdata\\local\\scoop\\apps\\nodejs\\.npm-packages which seems to be working, and needs some more testing.

Here's the modified Scoop package I'm using with NodeJS v0.10.28
https://gist.github.com/ntwb/2ff3a6872798cc48c51b

I haven't looked at doing this for Ruby yet, though a similar approach I believe is possible.

This is also the basis of what I'm looking into for #88 and MariaDB and a few other packages such as Nginx and Apache could also use some similar loving. Currently the recommended approach with Apache is to update the DocumentRoot inside conf/httpd.conf.

I think it would be cool if we can come up with a 'standard' approach across Scoop to handle separating the binaries and data in packages such as the ones mentioned above. It's all just been a time thing, I haven't had any of it to test and try all the these things :(

[lukesampson]

How about if we add something so that a manifest can mark directories (e.g. node_modules) within the app's install directory that travel between versions? Scoop could automatically move these directories whenever you update or reset an app.

So this would be transparent to the app, and you wouldn't need extra config for each app to tell it where the shared directory is—the packages would just be in the default place relative to the whichever version of the app you're running.

I like your workaround for NodeJS @ntwb. But like you say, if there was a more standard approach across scoop that might make it easier to make packages and less error-prone. If we called these travelling directories "package directories" we could even assume that they need to be on your path, and further simplify manifests because you wouldn't need the extra env_add_path.

What do you guys think about this approach of travelling directories? Does it sound like it would cover the various scenarios?

[ntwb]

I like the concept of 'travelling directories', though at the moment I can't picture a common way...

Taking NodeJS for example, the root folder \AppData\Local\scoop\apps\nodejs\0.10.28 for an 'out of the box' scoop install nodejs looks like the following:

(Where 'grunt' is a globally installed NPM module it is installed in the root alongside the main NodeJS binaries application)

dir  -> node_modules
file -> grunt
file -> grunt.cmd
file -> manifest.json
file -> node.exe
file -> npm.cmd

We could move all the files and folders not installed if we checked against a log of installed files and folders as applications future versions could add new files and folders.

Backups, I want to be able to include 'data' and exclude 'applications' for packages in my backups. For example, Nginx, Apache, Vagrant etc I'd backup the data folders and not the application. The folder location should be a permanent folder location so I don't need to reconfigure my backup when I update.

Back to how I can't picture this, maybe I am now able to after writing the above, we have two types of packages:

Firstly NodeJS, Ruby, Git, SVN, Mercurial, PHP, Python, VIM etc we can use .dotfiles for much of the configuration, typically none of these require any data to be backed up and can have roaming configurations so you can quickly get your custom configuration up and running on any new machine or pull them into a machine you are working on from your own personal .dotfiles repo.
(Mine sucks and is really hacky at the moment and is why my public repo is terribly out of date)

Secondly apps such as Nginx, Apache, Vagrant, MariaDB would work well for 'travelling directories'.

So both groups could have 'travelling directories', the second in particular and adding more '.dotfiles' support to Scoop https://dotfiles.github.io/ would allow the first group to store their configs and data in their own 'travelling directory'.

We could then extend Pshazz/concfg where appropriate with Windows/PowerShell compat repo's similar to fresh and dotphiles to have awesome configs to import and also sync our own personal .dotfiles repos with full Scoop support.

[lukesampson]

@ntwb Your dotfiles approach seems like a fairly sophisticated solution—I think I understand the basic idea, but I'm not sure I understand the implications fully and I'm not really sure what the implementation would look like. I'd like to get a simpler solution for people who just want it to work and don't mind too much where the packages are installed.

To help flesh out the travelling directories idea, I decided to try writing the documentation first, describing the very basic algorithm (if I can call it that) for travelling directories, and how they might work with Ruby and NodeJS specifically.

I figure if the documentation makes it clear enough how to go about using travelling directories, then we're in a good place to start implementing this approach. If you have time, I'd really appreciate it if you can have a look at the documentation and let me know what you think.

So I'm kind of avoiding the whole dotfiles thing because I can see it being fairly tricky to come up with a solution for each app, whereas with travelling directories you just say, "when people update, keep this directory and that one too".

[ntwb]

Apologies it took so long for a reply ;)

Alrighty, I'm with you, the docs and code make sense and the best way forward now will be to give it a whirl. I'm on a bus right now and am out for the night but will merge in your changes when I get the chance and see what happens.

As to potential issues:

• 'open' files, theoretically that would be a situation we would already be facing before moving down the 'travelling directories' path. Thus you'll need to close any opened files and/or stop any services that have files that will be required to be copied (Primarily I'm thinking databases and web server content here)

That's all I can think of at the moment, will no doubt come up with a few more.

[kodybrown]

Just my two cents, but I think it should copy the files/directories. Maybe 'copy' could be an option when calling update. Something like scoop update <pkg> --copy-data or something, while the default is to move the data.

This is valuable to me, because often I find that my plugins/extensions/etc will not work on a newer version of the package (Python, Ruby/Gems, etc.) and so I have to go back to the old version. In this situation, if the data files/directories were moved, that would mean that some of the data has been updated and some has not, making it difficult to go back.

Still, my thinking may be biased though, because I'm hoping to convince Luke that enabling basic version management in scoop is a great idea! 👍 ;-)

[jkrehm]

I like the explicit --copy-data option because it does give you more control, especially if the version change is large and you know there'll be incompatibilities.

The node.js implementation specified in the prototype docs won't work because npm itself is a node module and resides in node_modules. So the directory will ALWAYS exist, and we won't want to move the npm files.

npm's use of the AppData\roaming\npm directory really makes sense for this stuff. npm gets installed in the node_modules folder in the nodejs install directory, and then everything else (global modules) get installed in the roaming directory. What we need is a way to add such directories to the environment path.

With regards to permanent backups, I don't want my configuration getting overwritten by a new version. When I update, I use WinMerge to compare the versions and update the new one with my stuff. Is it possible to use junctions to symlink such configuration folders to a standard location? That way you don't have to update backup scripts, but the previous version isn't blown away, either.

[deevus]

Excuse me if I'm simplifying this too much, but why not store data in scoop/data/app? This way we don't need to have a concept of travelling directories. In most cases it wouldn't need to be on the path and can be specified in config.

[lukesampson]

I guess that would work for some programs, but how would we do the config?
Powershell scripts in post_install? I'm thinking this could get quite
complicated for maintainers, although it might simplify things from our
side. I think the maintainers' experience is more important to simplify.
On Fri, 7 Aug 2015 at 11:15 am, Simon Hartcher notifications@github.com
wrote:

Excuse me if I'm simplifying this too much, but why not store data in
scoop/data/app? This way we don't need to have a concept of travelling
directories. In most cases it wouldn't need to be on the path and can be
specified in config.

—
Reply to this email directly or view it on GitHub
#126 (comment).

[MPLew-is]

I definitely support having some process for moving data and configuration when updating, even if it's just a post_update script that receives the old and new app directories as parameters, and can then do whatever is needed to move data before starting the new services. I can currently kludge together a clone of this by having the post_install script look for the previous version of the app and execute the update commands if found, but having some way of making that process easier would certainly be nice.

I think that the app data should be kept locally to the version, since it's entirely possible that an upgrade will seriously break something if not done correctly, and copying the data from one version to another means there will at least be a backup if things go wrong.

[lukesampson]

@MPLew-is would you like to claim this and try to implement the travelling directories proposal? There is already a travel_dir function which you can use.

[MPLew-is]

Sure, I'll claim it. Looking over the proposal though, I have a few proposed modifications:

1. The addition of "traveling files". I know for PHP, for instance, the php.ini file is in the root directory, and should travel to the new installation.
2. The addition of a way to force the copying of directories. For example, in the current MySQL app manifest, the MySQL data directory is initialized on installation, but should be overwritten with the new data, as the directory is effectively "empty" when initialized.
   • I don't know what the best way would be to implement this on the manifest, so any input would be appreciated
3. The addition of a post_update script that would run after the directories/files have been moved, as a catch-all to handle anything more complex (for instance, upgrading MySQL sometimes requires the running of a script to bring the old data into compliance with the new version of the server).

Those are just the first things that popped into my head as to what more I might need to use this functionality, so please feel free to comment.

[lukesampson]

1. The manifest travel_dir array could become just travel and files could be legal there in addition to directories.
2. The intention was that only sub-directories of a travelling directory that already exist at the destination would be ignored when copying them, but not the travelling directory itself. So if it's "travel": [ "data" ], and a new data directory is created empty during the update, it should still be updated with the contents of the old data directory. Not sure if the implementation of travel_dir supports that, but it looks like it might. Does that sound okay?
3. How about adding a note to say "run path/to/mysql/update.bat to update your data" or whatever? I would like to limit the scope (and complexity) as much as possible at first. But we could definitely revisit adding post_update once everything else is working smoothly.