Mutagen Helper (file based session management app for Mutagen)

[Toilal]

Sorry if it sounds like spam, but I've just released 1.0.0rc3 for Mutagen Helper that now supports many handy features.

Would you consider adding a link to this tool inside README.md or on mutagen website ?

My use case is the following, that's the reason why I wrote Mutagen Helper, but there are other ways to use it that could best match other developers.

I have a directory C:\projects where resides a bunch of git repositories.

I need to setup synchronisation sessions for some of them, at some time, but never all at the same time. I'll maybe spend 1 hour on a project, and then 2 hours on another, and so on. I need to quickly start and stop sessions associated to each project to keep good performance.

Synchronisation session are always established to the same beta, but with an additional "path" matching the project name.

C:\projects
|- mutagen.yml # Configuration file
|- <project1> # will synchronise to <beta>/project1
|- <project2> # will synchronise to <beta>/project2
|- <project3> # will synchronise to <beta>/project3
|- <project4> # will synchronise to <beta>/project4

mutagen.yml configuration file contains the following.

beta: 'vagrant@192.168.1.100:/home/vagrant/projects'
auto_configure: True
options:
  scan-mode: accelerated
  sync-mode: two-way-resolved
  default-file-mode-beta: 664
  default-directory-mode-beta: 775

mutagen is freshly installed and available in PATH, there's nothing more to configure.

Then, you can create sessions for all project directories with the up command.

mutagen-helper up

And control running sessions with list, pause, resume or down

mutagen-helper list # List all sessions
mutagen-helper down project4 # Terminate sessions for project4
mutagen-helper pause project3 # Pause sessions for project3
mutagen-helper resume project3 # Resume sessions for project3
mutagen-helper down # Terminate all sessions

Mutagen v0.9.0b2+ performances are great, and it makes mutagen the BEST solution to share files in a remote development environment. I was looking for a good solution for about 3 years, i'm pretty sure I have tested all of them, so you can trust me. It should be used by anybody who daily runs Docker Linux containers on Windows or Mac.

Please note that I didn't found any issue related to scan-mode accelerated (I'm running Windows 10 on alpha, and Ubuntu 18.04 on beta).

[xenoscopic]

@Toilal This is definitely not spam, this is super awesome! I've added a link in the README.md with a short description, but please let me know if there's a better description that you'd prefer. I'll also definitely add a link on the website soon - I'm in the process of rewriting and restructuring the documentation right now.

I've also created a Spectrum chat for Mutagen, which is currently empty, but if you'd like to join and inaugurate the General and Sharing channels with a link to Mutagen Helper, that'd be great!

I wanted to let you know that Mutagen is going to be adding a new feature in v0.10.0: network forwarding. I'm not sure if this is something that you would want to integrate, but it would probably fit in nicely. It works mostly the same way as synchronization (except that it's unidirectional):

mutagen forward create <source> <destination>

It supports TCP and Unix domain socket forwarding, so you can do crazy things like:

mutagen forward create tcp6:[::1]:8080 docker://testcontainer:unix:/var/run/some.sock

It's still a little experimental, but it mostly re-uses code from the synchronization infrastructure, so I'm hoping it will stabilize quickly. I should have a beta release tagged very soon with these new features along with documentation, so I'll update this issue once that's available.

As a result, I've restructured the CLI slightly. I've moved the create, list, monitor, flush, pause, resume, and terminate commands under a sync command, so synchronization now looks like:

mutagen sync create ~/some/path user@host:~/other/path

Basically nothing has changed except that the commands now live under the sync command. In order to keep backward compatibility with your tool, I've kept a hidden copy of these commands under the root, so you can still do:

mutagen create ~/some/path user@host:~/other/path

I'd like to remove those hidden commands at some point, but I'm willing to leave them around for a few versions if necessary. Let me know if there's something I can do to help make that transition smoother.

One other thing worth mentioning is that Mutagen v0.10.0 is going to abandon TOML and switch to YAML for configuration. Ideally I'd like to reserve the names mutagen.yml and .mutagen.yml for Mutagen itself in order to avoid confusion. Would it be possible to switch Mutagen Helper to look for mutagen-helper.yml? Sort of the same way that Docker Compose uses docker-compose.yml?

Thanks again for your awesome contributions over the years!

---

P.S. Sorry for my radio silence - I'm hoping that will be the last time that I'm out of touch for so long.

[Toilal]

Sorry for the late reply, I was on vacation and just read this message.

I'll rename the configuration of mutagen-helper no problem.

All orchestration stuff from 0.10.0 is great and it makes mutagen-helper almost useless now (and this is really a good thing !).

There are still some use case though where mutagen-helper is still useful, so i'll keep using it for now, but with few enhancements to the mutagen orchestration feature, it could be dropped. So here are my feature requests:

1. It should be possible to name a project, so it's then possible to start and stop a project from any cwd. In mutagen-helper, the project name defaults to the name of the alpha directory, but it can be overriden with project_name inside the configuration.

2. It should be possible to perform mutagen project commands from a directory that doesn't contain any mutagen project configuration file. Default alpha value should be the current working directory (.). Default beta value could be provided by mutagen global configuration, where the project name is automatically appended to.

Example:

If you cwd in C:\workspace\my-project and mutagen is globally configured with beta: 192.168.1.100:/home/vagrant/projects, then the synchronisation session created by mutagen project start is C:\workspace\my-project <=> 192.168.1.100:/home/vagrant/projects/my-project (note that the project name is appended to beta).

Maybe beta for this configuration is confusing, it could be beta-base or something like that.

Maybe it should be a pattern too, like 192.168.1.100:/home/vagrant/projects/${project_name}.

[xenoscopic]

I'll rename the configuration of mutagen-helper no problem.

Thanks, I appreciate it, especially since it's a post facto request.

All orchestration stuff from 0.10.0 is great and it makes mutagen-helper almost useless now (and this is really a good thing !).

I think it can be really helpful to have two options - it helps to enforce interface boundaries and keep one option from becoming too tightly integrated. A Python option is more hackable as well, so it might be easier to play with ideas there and implement them in Go if they work out. Even just preserving the order of YAML keys in Go is really painful (Mutagen doesn't do it at the moment), whereas in Python it's trivial via OrderedDict.

It should be possible to name a project, ...

That makes sense to me. At the moment, projects are sort of a "faux" Mutagen object, just encoded by a UUID in a lock file. At some point, they'll become "real" objects in the Mutagen daemon like sessions, and I agree that having names (or labels, or both) to address those projects will be helpful.

It should be possible to perform mutagen project commands from a directory that doesn't contain any mutagen project configuration file. ...

That will probably require more thinking. If it's only creating a single synchronization session, then maybe it should just be a default behavior of the mutagen sync create command rather than part of the project orchestration? I worry that it might be something really specialized for a particular workflow. Does Mutagen Helper do something similar at the moment that I could look at as a reference?

[xenoscopic]

I just wanted to say thanks again for the work you've done on this. I still think it's important to have multiple orchestration tools that can be optimized for different workflows.

It's also helpful to have an implementation that's in a flexible language like Python, because it makes it easier to experiment with new designs for orchestration. I think it would be really helpful to prototype some of the ideas from above.

I'm going to close this to tidy the issue tracker, but (for posterity) I'd like to reference issues #38 and #57 since these are really the biggest barriers to improving the integration experience for external tools like Mutagen Helper and exploring some of these other ideas.