-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Systemd 'conversion' & GAT updates #63
Comments
+1. For that to happen, the release testing team must be made up of at least 1 admin and at least 1 dev who is sufficiently comfortable with gravity. That, and a revision of how we do release testing: we have a reasonably well-structured approach to testing end-user-facing stuff, but absolutely not admin-facing stuff. Maybe we should change that. |
Yeah, no equivalent to this.
No wildcard, but you can follow multiple with
I agree this is more annoying though, wildcard is a good feature worth adding if it's not too much trouble. It should apply to many of the subcommands like
We default You could also configure systemd to run You can also set In general with journalctl, personally, I have always been less comfortable having to use journald over flat log files, but I readily admit that is mostly just me being crusty and antiquated.
Yeah, no individual controls here. I am not sure how much complexity we would need to add to implement this via the supervisor process manager and if we'd be wasting our time doing it over alternatives (see below).
Conversely I always find it difficult to find older logs (if they are even available) with journald, but again, this is mostly my fault. We should probably however at least default the rotation option if flat files are used.
Still has to be installed somehow (e.g. via the role) but some of the stuff that we include to set it up to run in the role no longer needs to be done if you run it with gravity. However, #55 is important and needs to be addressed.
Same. Installation is still external to gravity.
I largely agree, although one thing I was never satisfied with was that everything was disjointed and basically all done via a bunch of disparate ansible roles that you figure out how to use via various trainings that have to be put together in the right order. Gravity sticks everything in one place to try to reduce the number of pieces you have to assemble yourself. With the addition of celery and removal of mules this was important, since you could no longer start and run a fairly featureful Galaxy with a single command. Although most Galaxies deployed using the training method in the past few years have used systemd/journald, they also ran mostly from a single service unit, something that would be broken with the addition of all the new processes, and had to be addressed in one way or another. As you said, you don't have the bandwidth to reimplement as systemd units, and we chose not to take that route (yet) either since it would be a solution for one specific case (production deployment with root privs on Linux). All of that said... Gravity was built to be fairly modular and agnostic with respect to the process manager, and so with some effort it would be possible to implement a systemd process manager as an alternative to supervisord. One issue is Gravity expecting to be run as the Galaxy user - and while you can run Galaxy from systemd user mode (The galaxyproject.galaxy Ansible role supports it now, for example), many admins would want it to work at the standard system/root level. This might mean creating a standalone Gravity venv for systemd installs rather than using Gravity from the Galaxy venv. And yes, absolutely, we need more hands (especially mine, since I am one of the folks most intimately familiar with all of this) on the updating... I have been underwater all of this week and I know we're running out of time before the training, but I am still planning to make a massive push to get everything updated. |
@natefoo thanks for the considered response, I appreciate it and agree with basically everything you've said, especially
And completely sympathise that having it all in one place (somehow) makes a lot of sense. This always was absolute rubbish, the mix 'n match of unofficial roles. Hopefully the collections and new ansible stuff you're working on helps there, I'd love to have all of the roles in one place with example playbooks. systemd as an alternative process manager makes this a lot more interesting to me, and gives me a lot of hope for this then. I would not mind installing it twice, that starts to feel like the best of both worlds; gravity as a single synchronized point, and systemd for all of the aforementioned features.
rest, relax then :) After discussion with @jdavcs, @Slugger70, @cat-bro, based on our lack of knowledge about gravity (, and the lack of a release) we've decided to re-use the old videos, staying on 22.01 for this GAT. We're going to use the archival training materials for the first day. |
should we write a journalctl training module for our old crusty folks? 😃 There's bound to be more who aren't comfortable with it yet, and lots of nice advanced features we can discuss. |
Yes, let's call it "Helena teaches Nate how to use journald" and then I will forget it all the next day anyway as I grumble about how UNIX was supposed to be all about files. |
Work is underway in my systemd-pm branch. Gravity has always had support for managing multiple Galaxy instances, but most of this is hidden from you if you only have a single instance. With supervisor, when you have multiple instances we use supervisor's group feature to keep the service configs separated. This will need to work differently with systemd. Here are my ideas: 1. Use systemd service instances We could in theory use systemd service instances (
So your venv always has to be at 2. Use service instances but Gravity writes out an environment file The service instance file would look like this:
Gravity would write out
This allows each instance to store its components wherever, but requires Gravity to keep a separate file up to date, which might be a bit more obtuse to admins. 3. Don't use systemd service instances This is the "simplest" method, you just end up with a bunch of service units for each instance of Galaxy, named like:
Conclusion Whatever we do, I was planning to do the same thing in systemd when there is only a single instance configured as we do in supervisor: hide all the extra instance stuff. So in the case of a single instance, you'd have just Since multiple instances is likely a not-often-used feature, I am not going to worry too much about this, but wanted to throw it out there in case you (or anyone else) had a better idea. |
Some thoughts on my own options.
I did discover one additional option that does appear to work:
The service instance unit file (e.g. # cat galaxy-@work2.service.d/40-gravity.conf
[Service]
User=nate
Group=nate
WorkingDirectory=/home/nate/work/galaxy2
Environment=GALAXY_CONFIG_FILE=/home/nate/work/galaxy2/config/galaxy.yml
# cat galaxy-gunicorn@work2.service.d/50-gravity.conf
[Service]
ExecStart=/home/nate/work/galaxy2/.venv/bin/gunicorn 'galaxy.webapps.galaxy.fast_factory:factory()' --timeout 300 --pythonpath lib -k galaxy.webapps.galaxy.workers.Worker --workers=1 --config python:galaxy.web_stack.gunicorn_config --pid /tmp/galaxy.pid That said, although it's more "systemd-ish," Gravity still ends up managing a bunch of unit files, at least one per service per instance, plus the top-level service instance config for each service, and probably a target unit for each instance (which I haven't even tested yet). In a non-instance setup you're going to end up having fewer unit files, just |
I have a strong strong preference for 3, 4 is not my preference, multiple files is more headache for admins to mentally resolve into what the unit actually looks like. |
Perfect, this will also be the simplest. Thanks!
Agreed, although TIL |
Yeah, those are nice (but then still have to figure out if it's overridden, which file, and blahhhh). My personal opinion is systemctl cat should be equal to cat (except in case of distro managed units), but that's probably a weird opinion only I hold. |
My systemd-pm branch was getting a little unwieldy and so I broke it up and refined it in more focused chunks. #69 and #72 get us most of the way there. |
The rest is in #77. |
Still need the ability to configure limits as you can in the Ansible role, I suppose. |
Limits would be ace! |
I think this is all taken care of now but if not feel free to re-open or spin anything remaining out into their own issue(s). |
hey y'all. I'm .. struggling a bit, with the updates to gravity from systemd. Currently it feels like I'm losing a lot of things that I use on a daily basis, but maybe I've just missed the equivalent replacements?
journalctl --since "30 minutes ago" galaxy
, I don't think there's an equivalent, I used this daily to find problems that happened at a specific time point. I don't really want to go back to looking at timestamps and parsing them with my brain.journalctl -fu galaxy-handler-*
, I can't seem to use wildcards. I used this regularly since I don't know which handler was serving a request, and I wanted to check which one errored, easily.journalctl -fu galaxy
, period. Now I have to know the log location (or as the galaxy user, run/srv/galaxy/venv/bin/galaxyctl --state-dir /srv/galaxy/var/gravity/ follow handler gunicorn
). This feels like a major downgrade from the sheer simplicity of systemd (yes I said it.)logfile_maxbytes
but I liked my old log rotation.journalctl --vacuum-size=500M
(great for removing old stuff when in a VM with limited space!) orjournalctl --vacuum-time=2d
(especially nice for deleting old logs, and being sure we aren't holding GDPR type data for too long.)Then there's some open questions for me
Pro: I know this would put all of us on a more standardised platform, and a lot closer to what main does (which feels good in theory!!). I know systemd isn't always well received, and doesn't work as well for "laptop" users, I understand/get/like the portability of supervisord, especially for all the small/laptop users who are replacing
run.sh
(and main.)Con: But I think bigger sites are really getting a lot of benefits from it that we really like. Many small galaxies started in the past 3-4 years all use systemd, journald. All of these admins will have to update what they're doing.
I guess I'm asking:
The text was updated successfully, but these errors were encountered: