Name sessions?

[128keaton]

I am in the process of moving from docker-sync to mutagen for my docker setup. It would be fantastic if we could pass a name to the mutagen create command. For instance, running mutagen list returns multiple syncs for my multiple containers (as I have different ignores for each container):

--------------------------------------------------------------------------------
Session: 6968348a-34a8-4c72-a6c3-4e40fb4bf668
Alpha:
        URL: /Users/keatonburleson/Documents/webapps
        Connection state: Connected
Beta:
        URL: docker://0d98cf8161da2e5911a1fc6af2921bb4bf5f09c32c5bfe6ab4fefc708489cc85/var/www/site
                DOCKER_HOST=
                DOCKER_TLS_VERIFY=
                DOCKER_CERT_PATH=
        Connection state: Connected
Status: Waiting 5 seconds for rescan
Last error: beta scan error: unable to probe root Unicode decomposition behavior: unable to create test file: unable to create file: permission denied
--------------------------------------------------------------------------------
Session: 344fe27e-cfcb-4240-a34b-aeac34a54918
Alpha:
        URL: /Users/keatonburleson/Documents/webapps
        Connection state: Connected
Beta:
        URL: docker://www-data@b18061a9c8f410cfa97f7b5f0b26b6eb6f7ade2033854037b5694aaf783fff37/var/www/site
                DOCKER_HOST=
                DOCKER_TLS_VERIFY=
                DOCKER_CERT_PATH=
        Connection state: Connected
Status: Applying changes
--------------------------------------------------------------------------------

Having the sessions named would also make it easier to clean up after stopping my docker containers.

Just a thought!

Ideally, mutagen list and mutagen monitor would return the randomly generated ID under Session: if the name is not passed, and if it is, a new ID: entry would show that randomly generated ID.

[xenoscopic]

Hey @128keaton, thanks for the feedback. Mutagen's current session "orchestration" (if you can call it that) definitely leaves a lot to be desired. It works well-enough for long-term synchronization sessions that you might set up and leave running, but it's an awkward fit for the world of ephemeral containers. At the moment, I'm exploring a few designs, and I'm not sure exactly where things will land, but I think there'll be an answer within about 2 weeks.

The question of named sessions (or at least controllable session identifiers) is something that has come up multiple times, both directly and indirectly (see, for example, issues #38, #43, #50, and, #57). Right now the primary proposal is to add "tags" (or "labels" - we haven't settled on a name) to synchronization sessions, which would allow users to more easily control synchronization sessions within the Mutagen daemon.

While I think that tags/labels are a decent solution, I'm also exploring the idea of just cutting out the Mutagen daemon entirely, and instead allowing direct invocation of Mutagen synchronization sessions, e.g.

mutagen session [options] <alpha> <beta> <data_directory>

This would create and run a session and store all of the ancestor/cache data in the specified directory. The lifecycle of this session would be controlled by the lifecycle of the process, rather than by the create/pause/resume/terminate commands. A session would be created when the process started, paused when the process terminated, resumed if the process was restarted with the same data directory, and terminated when the process was terminated and the data directory deleted. There would still be monitor and flush commands that would just take the data directory as their argument. Essentially the data directory would be the session identifier.

While this design is a radical departure from Mutagen's current design, I think it's ultimately a better fit for Mutagen's long-term future. It removes the impedance mismatch between Mutagen's daemon and session lifecycle model and the various orchestration tools/scripts that developers use. And it does so without losing much in the way of functionality for people who want to set up long-term automatically reconnecting sessions (they'll just start the mutagen session command in the background instead of the mutagen daemon command). If people later want to reimplement the Mutagen daemon, they'll be able to do so on top of this new interface (and I suspect they'll be able to implement much better "orchestration" interfaces on top of it). I think Mutagen's role should be restricted to just providing the synchronization algorithm/transport and that session orchestration should be left out of it.

Anyway, that's where I'm looking to take things right now, but it's a little premature to promise that it's exactly how things will look. I figured I'd just write this up to give you an idea and maybe ask for a bit of feedback.

[xenoscopic]

CC @Toilal, @IngCr3at1on, @phromo, @matthew-gill, and @divoxx in case they have any thoughts on this "daemonless" Mutagen design that I've proposed above. I know all of you have had interest in easier scripting for Mutagen in the past. I'll probably prototype it up anyway, but I'd be happy to hear your thoughts.

[IngCr3at1on]

Where would the agent go in this scenario?

[128keaton]

@havoc-io thank you for the fast response! That solution sounds pretty good. Cannot wait to see what we can do in the future (features like an initial copy would be fantastic)

[phromo]

I'm quite happy with the scripting I can do on top of the existing design. My compose scripts up and down commands use the alpha,beta pair as the unique key for a session and thus does not need an extra name. Having to daemonize myself would make my scripts unnecessarily complicated (I'd have to provide a a way to list, stop, start myself ...). So in summary, I don't see any problems with the suggested design, but don't see any major value either.

[xenoscopic]

@IngCr3at1on They'd probably still go in the same place (~/.mutagen/agents) on the remote, though I am experimenting with a few ideas around allowing them to be placed elsewhere. Getting rid of ~/.mutagen is tricky in general since there's are a few things stored in there (agent binaries, session metadata, ancestor archives, caches, staged files, etc.).

@phromo Thanks for the feedback, I think that's useful to hear. I think I'll work to prototype the proposed mutagen session command as an addition to the daemon rather than a replacement. I can still definitely imagine a use case for one-shot ephemeral sessions.

@128keaton Can you elaborate on what you mean by "initial copy"? If you mean starting with a base copy of the files (e.g. a snapshot baked into a container image from the latest Git commit or something), then you can actually do something similar now, though it takes a little bit of understanding of the synchronization mode that you're using and Mutagen's reconciliation algorithms. I'm happy to discuss via email or the mailing list if you're interested. If you mean something else, I'm interested to hear!

[128keaton]

@havoc-io in our setup, we have a composer image that is designed to boot, install deps from the composer.json file and shutdown. With docker-sync it works fine since its a shared volume available from the get go. I experimented a little bit and wrote something with unison, but that did not work well since mutagen does not cross the file system boundaries. Right now, what seems to be working okay (with a few issues) is another image with a python script watching for changes/creation of composer.json then running composer.

Ideally, it would work like this:

1. Create mutagen sync
   • Somehow everything would have to be preloaded so the composer image doesn't freak out
2. Start composer image
3. Wait for composer image termination
4. Finish syncing from beta---->alpha
5. Remove sync session for container.

[Toilal]

I wrote a python application mutagen-helper, just released today, that wraps mutagen binary and can help to manage your sessions.

Put configuration in a yaml file in a directory to sync, and run mutagen-helper up / mutagen-helper down to create / terminate sessions.

It supports naming sessions (optional), and you can create multiple sessions on the same "project" (=configuration file)

You should have a look :)

[xenoscopic]

Mutagen v0.9.0-beta2 brings label/selector support for sessions, which should effectively be a superset of the "named session" functionality requested. It's designed to be comparible with the syntax that Kubernetes uses (since it actually re-uses the Kubernetes implementation). Please feel free to give it a spin.

[xenoscopic]

Just a quick update to this issue: v0.10.0 adds support for session names, either through the -n/--name flag at session creation time or through orchestration configuration files. These names don't have to be unique across sessions, they're only used as a user-friendly mechanism for addressing sessions. More information can be found in the updated documentation.