Need to set up linux boxes to auto-update to stable packages

[larsbergstrom]

I currently run sudo dpkg-reconfigure --priority=low unattended-upgrades on new Linux machines by hand (it has a GUI popup). Presumably there's a better way to set this up to happen automatically.

[aneeshusa]

FYI, this is your third dupe of this issue 😉
See #207 and #102.

I will say that I've never been a huge fan of automatic updates (see also: left-pad), preferring pinning + manual updating (possibly with automated rollout), but this is probably safe.

[larsbergstrom]

@aneeshusa HA! Great catch. I keep using different words and never see it.

I tend to open these when I log into a machine and see "147 security updates needed," which always freaks me out. I'd personally like to have at least something basic that will get us security updates, even if it means we might left-pad our CI infra one day rather than leave machines unpatched or rely on manual updating.

[aneeshusa]

Something else that might help is auditing the packages that are installed; I just checked servo-master and I'm seeing around 800 packages. This is something that traditional CM systems like Salt aren't so good at. For us, a simple pkg.purged state with a long list of pkgs will do, but better solutions (longer-term) would be starting from a more minimal Ubuntu image, or using NixOS or a similar declarative distribution, where only packages you explicitly list are installed (and packages you unlist get removed).

[larsbergstrom]

I definitely agree! The servo-master machine is particularly bad, and I'm glad (though also terrified!) that @edunham and I plan to burn it down and rebuild with salt from scratch.

I believe that it's extra bad because in the dark, dark early days we all did our testing and part of our CI development directly on that machine. It's also survived major transitions (e.g., from the "bors" autolander to "homu," completely different methods of github events vs. polling tools, etc.).

Is the pkg.purged just supposed to have a list of all of the approved packages?

[aneeshusa]

Well, pkg.purged will ensure those packages don't exist, so we would need to list any packages that we don't want installed (e.g. extra packages in the base image we're using). It's not perfect, but it should do for now.

As I mentioned in #215, look into using Salt multimaster to manage the servo-master transition; I'd also recommend setting up the new gitfs based configuration on a new master and cutting over once it's confirmed working, instead of deploying it on the existing one.