Make VM journal volatile by default #8832
Comments
emanruse
added
P: default
|
Implementation detail if implementing this:
|
|
The oneliner was just an example of what the result should be and how to do it easily. Otherwise editing the conf file is the right thing to do.
I don't understand the relation to /usr/lib/systemd. Making journal volatile does not require a systemd unit.
|
|
Derivative distributions are not supposed to write into The appropriate journal configuration drop-in folder is: https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html systemd has always been good in providing drop-in configuration folders to make customization simple. I can almost guarantee if Qubes implements this, a drop-in will be used. That's why Qubes It makes no sense to debate that. That's what Qubes always did. So if you want to send a pull request, I would suggest |
|
Derivative distributions are not supposed to write into `/etc` if avoidable.
I don't know if Debian and Fedora templates are considered derivative distributions but Qubes OS already writes custom configuration stuff in /etc. Changing one line in a conffile should not be a problem. It would solve a problem.
|
|
No, /etc is for user changes, packages should avoid modifying config files there of other packages if possible. Anyway, if this change is to be made, it's clear how it should be done (via a drop-in file in /usr). What isn't clear, is whether it's a good idea to change it at all. A better solution might be making journal survive qube restart, instead of breaking it even further. |
|
No, /etc is for user changes, packages should avoid modifying config files of other packages if possible.
It is possible but the question is whether it is justified in the particular context/case.
Anyway, _if_ this change is to be made, it's clear how it should be done (via a drop-in file in /usr).
From FHS:
"/usr/etc Theoretically, that's another directory for configuration files. Virtually unused now."
Is that what you mean?
What isn't clear, is whether it's a good idea to change it at all. A better solution might be making journal survive qube restart, instead of breaking it even further.
Considering Qubes is a desktop OS:
- What would log persistence improve?
- How many users actually look at logs?
- How many users need to backup tons of logs (stored in each qube)? Will there be a way to exclude logs from backups?
- Considering also that, so far, logs have always been volatile in AppVMs, is there any negative experience from that? How many actual complaints are there? Any surveys at all?
- Considering it is easy to bind-dir /run/log/journal, in case this is really needed, is persistence by default actually necessary? For what particular purpose?
- Considering running many VMs means lots of logging, which of the 2 possibilities will be better for the default overall performance?
Maybe answering these questions will make it easier to decide.
|
No, see earlier comment by @adrelanos |
|
Perhaps journal should be persistent or non-persistent as per Qubes respecting upstream distribution culture, FAQ What is Qubes’ attitude toward changing guest distros?. If Debian has persistent journal by default, Qubes Debian would also have it. Respectively depending on what Fedora does etc. minor: https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html unfortunately does not seem to support
Log analysis in case of issues without complex instructions "1) make log persistent in Template, 2) reboot App Qube, 3) reproduce the issue, 4) check the log".
Only advanced users and developers. I am a developer but I am also a user. By putting more obstacles into my way, the less productive I get. For example, if a VM is slow to shut down or crashing, I need to look at the journal of the previous boot. ( But if extra steps are required then that takes extra time and extra mental load. Hence lowering productivity. Therefore, I am all for to make things comfortable for advanced users and developers to keep the ability to easily debug the system.
I guess not. That seems complex to implement to have Qubes backup parse the contents of a VM.
Performance will be negligible / non-issue / same as now because now all VMs are logging just that it gets lost after App Qube gets shut down. |
|
Perhaps journal should be persistent or non-persistent as per Qubes respecting upstream distribution culture [...]
Or maybe it is wiser to consider what Qube OS's specifics are and what defaults fit that. It separates system/user data explicitly, so if distro X changes its defaults tomorrow and starts flooding one's AppVMs with logs, or vice versa, there will be no flashing red lights to warn the user. Journal is system, not user data, so it doesn't belong in private volume (unless explicitly requested).
does not seem to support `ConditionPathExists`
In both Fedora and Debian, the default is Storage=auto (from man page):
"auto" behaves like "persistent" if the /var/log/journal directory exists, and "volatile" otherwise (the existence of the directory controls the storage mode).
Perhaps that could be used as a control flag (e.g. a bind-dir which advanced users have no problem with), so no conffiles need to be touched.
Re. the answers questions: They are about evaluating the average Qubes user's workflow, not about what you or I prefer. That's why the skipped question #4 is essential.
Without any actual statistical data, I can only remind that 90%+ of desktop users will generally abandon software which doesn't behave as they expect, rather than report bugs or analyze logs (or even know what a log is).
|
I would also prefer if Qubes would ship hardened by default, customized forks of Debian, Fedora or similar. (#8730) But that is not the case (yet?) and might be related to:
This is probably historic reasons in case there is no alternative yet. (I am writing "probably" to be extra careful to not speak for Qubes even though Marek already agreed with the configuration file location, I shouldn't assume to know the detailed reasoning behind it.) Elaborated below.
They are in my view and that should be explicitly documented (elaborated in #8730) but that shouldn't matter. I can rephrase this a bit better without needing to mention "Derivative distributions".
Better rewritten to: Why this push exists and references for that, documented just now:
It's a problem because this can later result in an interactive dpkg conflict resolution dialog. Why that is an issue is now documented here:
I will assume this is correct. Could easily be correct. In that case, it is important for the remaining 10% of users to make it as simple as reasonably possible to debug and report issues. Complex instructions "1) make log persistent in Template, 2) reboot App Qube, 3) reproduce the issue, 4) check the log" should be avoided. |
|
FWIW the alternative (preferred?) way of persisting logs from app qubes (and others) is logvm (#830). But that, besides implementing the actual feature (which is rather simple, and there are few PoC already), requires consideration of memory usage - yet another always running qube is IMO too much with their current memory usage. We need to make them lighter first (for example #7159 ). When logvm gets implemented and enabled (including the above lightening prerequisite), setting journal to volatile will be the obvious change to make. But before that, I don't think it's universally a good idea.
At least in Fedora, the
With volatile journal, it can still be accessed until qube shutdown. Volatile means it's stored in memory only (AFAIR somewhere in /run), not discarded completely. There is likely a limited amount stored (few MBs?) but should be still good enough to get logs of something that just happened. The real behavior change (compared to the current situation) if setting volatile journal applies to the template itself and standalones only. But then, it might be useful to get logs from previous app qube startup (for example to get more details why it crashed) - which currently is not possible. Having persistent journal (via bind-dirs?) would solve this. But also probably should also be limited in size (the default is quite high IMO - 10% of the disk, which would be 200MB here in default config).
Yes, I agree packaged configuration (modifications) should be kept out of /etc, to not conflict with user modifications. Having both package and the user modify the same configuration file is complex and error-prone. It doesn't matter if that's derivative distribution or not - it applies to upstream distributions to some extend too (some go even further - openSUSE has a goal to have fully functional system running default configuration with no packaged configuration in /etc, so anything in /etc is in fact user customization). Sadly, not all applications support config drop-ins yet. And some do not support multiple configuration files at all (the only one is in /etc or user home dir). For those cases, we still need to handle it via editing config in place (or, sometimes preferably, avoid config change at all and consider alternatives). |
I meant to debug a VM crash which requires the log of the previous boot.
The number of past reboots could be limited. 1, 2 or 3 prior boots seem reasonable numbers.
Yes. |
|
If the templates are to be considered derivatives (which, IIUC, is not decided), there should be no conflict to resolve - the derivative should be free to act based on its own goals.
If the templates are not to be considered derivatives, the question is much bigger than the current issue, because Qubes OS has its own specific goals and approach which are not those of any Linux distro. So, instead of trying to conform to something upstream for the sake of... I really don't know what... it might be worth reconsidering the architecture as a whole. We can discuss further if you will:
https://forum.qubes-os.org/t/distributed-os-more-suitable-for-qubes-os/23684
Complex instructions 1), 2, 3...
For those ready and willing to report bugs and knowing what it means, these instructions are not complicated.
setting journal to volatile [...] I don't think it's universally a good idea.
Well, currently it is volatile and IIUC it has always been like that, and so far I don't see any actual complaints from _users_ (not from devs who spend their day in debugging and know how to enable it anyway). The current issue simply suggest to stop hammering on the SSD for no reason.
At least in Fedora, the `/var/log/journal` directory is owned by the `systemd` package. So, removing it by default isn't really an option.
How about linking it to /dev/null?
The real behavior change (compared to the current situation) if setting volatile journal applies to the template itself and standalones only.
FWIW, I never considered standalones for the current issue. To me they are what they are - standalones, handling their own stuff internally.
But then, it might be useful to get logs from previous app qube startup (for example to get more details why it crashed) - which currently is not possible.
Wouldn't a crash of a qube be logged into dom0 anyway?
|
That still means overriding a file/directory owned by another package. It will cause issues on updates.
You will see it crashed. Usually you will not see why it crashed. Anyway, I must admit that relatively low (as in - almost non-existent) number of cases when having such logging enabled by default would be necessary for debugging some issue is rather compelling argument to not enable journal persistence by default. It's needed only if:
FWIW, I personally use a different method to keep qube's journal persistently - enable logging to the console ( Maybe some middle ground would be:
|
|
Maybe some middle ground would be:
- disable writing journal to disk by default (at least in app and disposable qubes)
Also in templates, so qubes don't inherit logs which are not theirs.
- enable persistent journal when "debug mode" is enabled (checkbox in qube settings -> advanced tab); do it via bind-dirs, so logs can be browsed using standard journalctl tool
Sounds good to me.
|
The problem you're addressing (if any)
Considering the way Qubes OS works, journal persistence in client qubes makes no sense, as the volatile volume is volatile anyway. Tracking potential issues can still be done by observing the current boot journal, or explicitly, temporarily enabling journal persistence (in template or using bind-dirs). This is an advanced administrative task anyway, so it is not supposed to be done by users on a regular basis, i.e. won't hurt usability. Other distros (e.g. openSUSE) have volatile journal by default too.
The solution you'd like
In the template:
Reboot the template, then remove the old persistent journal from
/var/log/journal.The value to a user, and who that user might be
Moving journal to tmpfs would reduce unnecessary SSD writes.
The text was updated successfully, but these errors were encountered: