Create service abstraction layer (proposal, prototype)

[offlinehacker]

Idea

Let's abstract services, so they can be used independently by different process manager like supervisord, docker,... and not only by systemd.

Preface

I really like nixos and module definition schema, and i really like to write nixos services. As a developer i want to make these services portable betwene systems independently of process managers if that is possible for speciffic service. Current systemd interface is not right approach and only results in hacks to support its interface(like https://github.com/kiberpipa/nix-rehash/tree/master/nix-services).

Problems with using systemd interface

Problem with using systemd interface as a base for service definitions for other process managers is that service definitions use all features of systemd, even that other process managers do not support them. There are not sufficient nixos options to support lack or presence of if:

• process manager can handle forking services
• process manager handles logs
• process manager handler logs from syslog
• ... your use case ...

Implementation

Prototype implementation present in this pull request wants to solve this by abstracting services out of nixos. The reason for abstracting out is that services will not be only used in nixos, but also in other scearios with other process manager and with abscence of nixos on different systems, distros, process virtualization solutions like docker.

Because of lack of better idea i defined sal(service abstraction layer) interface abstracted into new service directory tree. There, service options/configurations are defined and process manager options are present. Real implementation of this schema is present in nixos systemd, which uses sal as a base for services.

I should mention that proposed solution is not usefull for all services and is not ment to replace systemd interface, which is by the way really nice and has a lot of features.

Alternative implemenations

More straightforward solution would be to make something like users.services interface, for defining services. This has a downside, and does not take in account which features different process managers support.

I would really need this now

I want to ask you, whether you like the proposed implementation or if you have any better idea how to implement this. I would really like to solve the problem, because i need it to make user services and docker implementation using this or similar interface. Without this i can only make hacked implementations(that people use and depend on), like https://github.com/kiberpipa/nix-rehash/tree/master/nix-services, that i later don't want to support, because of hackiness involved.

So please review the prototype and please propose better implementations and let's make nix services as portable as possible :)

@edolstra @rbvermaa @nbp @iElectric @lethalman @wmertens

[bjornfor]

How are service dependencies specified in this new Service Abstraction Layer? Like systemd "wants" and "after".

[offlinehacker]

Currenlty they are not, i am not sure that's really a part of this layer. Different process managers have different ways for dependencies. In case of docker you would run multiple containers for each process, and sometimes you run service on many instances and do load balancing and other services depend on it.

I think it's up to systemd, supervisord, docker to define dependencies in their own way.

[matejc]

+1

[7c6f434c]

Very nice. A selfish question: maybe there should be a separate mongodb.configs attribute set so we can easily get everything considered a config file? It is sometimes helpful for debugging, and specifyng it into a large service refactoring gives us a chance for a uniform interface.

[offlinehacker]

@7c6f434c yeah this is only one use case, this is not such an easy problem. I guess separate service definitions for user services could be a solution, but well then we are already talking about service abstraction layer.
I think sepparation is needed, so people will have clear idea what they are dealing with and know limits when writing services, currently everyone is "happy" with systemd, but on the other hand are using my https://github.com/kiberpipa/nix-rehash user services, which i'm not willing to support in such hackish way.

Honestly i'm quite limited with systemd, because i can't write services for other process managers and it hurts everytime i write service i could use elsewhere (user services, docker, osx,..).

I'm still waiting for response @lethalman @wmertens @edolstra @rbvermaa @nbp @iElectric? I would like to solve this so i could continue develop nix user services and docker services.

[7c6f434c]

@7c6f434c yeah this is only one use case, this is not such an easy problem. I guess separate service definitions for user services could be a solution, but well then we are already talking about service abstraction layer.
I think sepparation is needed, so people will have clear idea what they are dealing with and know limits when writing services, currently everyone is "happy" with systemd, but on the other hand are using my https://github.com/kiberpipa/nix-rehash user services, which i'm not willing to support in such hackish way.

I wonder what will be the next thing to break — broken logging of
service startup errors was considered minor, impossibility of anything
like startx was said to be a niche problem (there I am, debugging small
problems related to fully hand-written boot sequence), but bugs with
swap in fstab seem to be generally annoying.

Honestly i'm quite limited with systemd, because i can't write services for other process managers and it hurts everytime i write service i could use elsewhere (user services, docker, osx,..).

And then, sometimes the service needs just a start script and stop
script — without any management (having proper service management is not
always cost-effective).

By the way, is preStop script implied to be capable to properly tell
service to stop?

I'm still waiting for response @lethalman @wmertens @edolstra @rbvermaa @nbp @iElectric? I would like to solve this so i could continue develop nix user services and docker services.

Well, I am so not OK with SystemD that I will be willing to join you
even for a fork-first-merge-later scenario…

[lucabrunox]

The original dir was /var/db.

[offlinehacker]

Ok, i will that, it was mostly as example, i have to make this interface better anyway.

[lucabrunox]

Also, I think this is the best time for thinking about multiple-instances of services? Instead of having services.mongodb have services.mongodb.<instance>, where the default one is services.mongodb.default.

[offlinehacker]

@lethalman can you explain multiple-instances of services in more detail?

[nbp]

inherit (config.service) description environment; and so on ;)

[offlinehacker]

Yup, makes sense ;)

[nbp]

Having an abstraction layer for services is a good thing, as this brings more competition between multiple systems and reduce the cost for changing from one to the other without as much burden as other distributions (* -> systemd).

The current prototype is not yet satisfying from my point of view. One of the issue I see is that the MongoDB module is no longer 1 module, but 2 modules. One defined as part of "sal", and the other defined as part of NixOS.

The one defined in "sal" contains most of the logic for starting a service, but none of the logic for making it secure when we are capable of expressing the need of a new user associated with the service. The user name is not always something which appear last in a configuration, and as such I do not think it should be a second citizen option.

The other part are the systemd option definitions, why do they have to be out-side the first MongoDB module? I guess I should put mock-up system as part of the module logic … and not as part of any mock-up module.

[lucabrunox]

My idea of multi instances is to have multiple instances of the service :P Instead of e.g. one nginx httpd, you can define multiple nginx httpd using different versions for example. E.g. pgsql 9.3 and pgsql 9.4.

But forget that, it's not tied to this PR, I'm going to do that separately. Though I'd like these new services to be multiple-instances aware as possible.

[lucabrunox]

@nbp honestly, changing init system is not a burden even in the current stage. upstart -> systemd translation is straightforward. But I feel adding an abstract layer makes services more "standard".

[lucabrunox]

About systemd services, those should be more automatized starting from the sal services. Otherwise it becomes annoying to write services. Using an helper function like mkSystemdService = salService: otherOptions: ...

[offlinehacker]

@nbp

The current prototype is not yet satisfying from my point of view. One of the issue I see is that the MongoDB module is no longer 1 module, but 2 modules. One defined as part of "sal", and the other defined as part of NixOS.

I can agree here.

The one defined in "sal" contains most of the logic for starting a service, but none of the logic for making it secure when we are capable of expressing the need of a new user associated with the service. The user name is not always something which appear last in a configuration, and as such I do not think it should be a second citizen option.

Ok we need some security definitions. But i have to think here, managing users will not be everywhere(os x, user services). @nbp @lethalman do you have any idea how we could do this without defining users in sal?

[offlinehacker]

@lethalman

My idea of multi instances is to have multiple instances of the service :P Instead of e.g. one nginx httpd, you can define multiple nginx httpd using different versions for example. E.g. pgsql 9.3 and pgsql 9.4.

I would like to work on that when we solve this is done in some way.

About systemd services, those should be more automatized starting from the sal services. Otherwise it becomes annoying to write services.

Yes, but user managment, this is not part of sal services. Sal has to provide required deffinitions for security, so systemd will be able to define users. And yes in that case i should also add service dependencies in sal, but that's not such a big problem i guess.

[7c6f434c]

My idea of multi instances is to have multiple instances of the service :P Instead of e.g. one nginx httpd, you can define multiple nginx httpd using different versions for example. E.g. pgsql 9.3 and pgsql 9.4.

But forget that, it's not tied to this PR, I'm going to do that separately.

It is a very nice feature, and this PR is just small enough to try to
spec in some sane behaviour for multiple instances from the very
beginning.

But I guess it is OK if it is clear how to add this when/if this is
merged. Of course, after the merge it is a good idea to collect such
improvements (I guess multiple instances is not the only one) for some
time before actually converting any significant amount of code to the
new style.

[7c6f434c]

@nbp honestly, changing init system is not a burden even in the current stage. upstart -> systemd translation is straightforward. But I feel adding an abstract layer makes services more "standard".

Well, a global transition with the minorities losing is one thing (and
easier to maintain); sharing the same code base in a way that makes
trivial for a part of users to switch without anyone noticing is another
feature.

I looked at just extracting service definitions from our current NixOS
service modules: it is not straightforward without a rewrite of the
NixOS code (and as this PR is a smaller change than I would spec, so I
use ad-hoc code for transition time and would like this change as a long
term solution).

[7c6f434c]

The one defined in "sal" contains most of the logic for starting a service, but none of the logic for making it secure when we are capable of expressing the need of a new user associated with the service. The user name is not always something which appear last in a configuration, and as such I do not think it should be a second citizen option.

About user name being second class citizen: in a sense, it is a second
class citizen. It is not something to be reasonably used on all systems.

Maybe this could be handled less radically, but the distinction is
there.

The other part are the systemd option definitions, why do they have to be out-side the first MongoDB module? I guess I should put mock-up system as part of the module logic … and not as part of any mock-up module.

Well, there will always be modules that can be made to work only inside
NixOS-on-SystemD (most of them are not strictly service modules, but
still currently implemented as SystemD units).

Either these have to be made pseudo-SAL-services with little plans for
portability and some extra effort to access unportable features in a SAL
compatible way (we need to make it easier to use portable features than
unportable ones for normal services, so access to unportable features
has to come as an afterthought),

Or NixOS will have its own service list with some of the services being
explicitly wrapped from SAL.

(I currently care only about SAL and don't care for NixOS-with-SystemD
anymore, so I have no idea which option is better)

[7c6f434c]

About systemd services, those should be more automatized starting from the sal services. Otherwise it becomes annoying to write services.

While we talk about annoyance...

I guess we need to consider templatability of both SAL part and SystemD
include-sal even more than their brevity.

Writing services from scratch is already more annoying (relative to just
writing a start-script and stop-script with Nix ${}) than writing
packages (relative to a build script). For packages I still prefer to
start from a ten-line template and fill in the blanks. Maybe we should
have an official fill-in-the-blanks template for service?

[offlinehacker]

Thanks for all the ideas so far, i'm working on something that would try to cover everything from security, dependencies and data(directories) in a as general way as possible while not trying to overcomplicate everything.

[wmertens]

I tried to abstract the SAL. I started by reasoning that systemd is probably the most complete process supervisor and as such all other supervisors will implement subsets of what systemd does. The systemd manual says:

A unit configuration file encodes information about a service, a socket, a device, a mount point, an automount point, a swap file or partition, a start-up target, a watched file system path, a timer controlled and supervised by systemd(1), a temporary system state snapshot, a resource management slice or a group of externally created processes.

Let's draw lines:

• In the SAL we are interested in "user"-oriented services that communicate via sockets, D-BUS or the filesystem, either one-shot or continuously.
• Multiple instances of the same service can exist, as long as they don't have conflicting resource use, e.g. socket binds, single-writer paths.
• SAL-defined services should have a configuration as input and an abstracted service as output.
• The SAL does NOT do:
  • service supervision
  • logging stdout/stdenv
  • userids, permissions, limits, timers, devices, mount points...
  • These are the domain of the service supervisor, e.g. systemd for NixOS, supervisord, or even simply a user running things manually.
• SAL use is optional, e.g. a NixOS-specific service can skip a definition in the SAL

Abstraction:
To keep the SAL abstract, the interface with the supervisor should be consistent. Systemd has these interesting options for services:

• Type: simple, forking, oneshot and a few systemd-specific ones
• PIDFile/GuessMainPID
• ExecStart, Pre and Post
• ExecReload
• ExecStop and Post
• The other options probably apply to systemd only and are used for ordering or communication setup between the service and systemd.

These options could be abstracted as follows:

• $out/bin/${salName}-(start|stop|reload)(|-pre|-post): wrappers for service actions. They should set the environment and configuration files for that instance of the service. If absent, the semantics of systemd apply, e.g. no stop means that the service will simply be killed.
• $out/pidfile: if present, a symlink to e.g. /run/service.pid. If absent, the service doesn't have a pidfile.

The SAL services should be functions that take a configuration and return an attrset with e.g.:

• salName: servicename + port or anything else that easily identifies the service
• wrapper: $out with the service wrappers
• parameters: All the non-specific systemd parameters: Type PIDFile BusName BusPolicy RestartSec TimeoutStartSec TimeoutStopSec TimeoutSec SuccessExitStatus Restart RestartPreventExitStatus RestartForceExitStatus Sockets(?) StartLimitInterval StartLimitAction FailureAction. Supervisors can ignore them if not implemented.

Note that Type can only be "simple", "oneshot" or "forking" and the other types will need to be overridden when using systemd. Furthermore, the time units can only be integer second counts.

So...

Relative to the current mongod example, the /services definition would be a function, returning a service wrapper for a given config. NixOS's systemd will need to know about PermissionsStartOnly=true.
A user could create their own mongod by defining a mongod config in .nixpkgs/config.nix and then installing the wrapper or using an environment definer that provides a supervisor for services.

Note that the config.* settings are still the domain of NixOS in this proposal. The SAL functions take direct configuration like the nixpkgs packages.

Thoughts on this?

[offlinehacker]

I started by reasoning that systemd is probably the most complete process supervisor and as such all other supervisors will implement subsets of what systemd does.

Systemd assumes to much and at the same time limits portability on anything but systemd. Docker has to deal with data volumes(where services store their data) and ports exposion betwene services, so let's think out of the systemd box(because it's really just a very limited box).

In the SAL we are interested in "user"-oriented services that communicate via sockets, D-BUS or the filesystem, either one-shot or continuously.

I have sockets now, will still have to figure how to deal with dbus, still thanks for reminder.

Multiple instances of the same service can exist, as long as they don't have conflicting resource use, e.g. socket binds, single-writer paths.

That's not something sal will deal with, sal is only about definitions. I think it's not too hard to add multy service support.

SAL-defined services should have a configuration as input and an abstracted service as output.

true

The SAL does NOT do:
service supervision
logging stdout/stdenv
userids, permissions, limits, timers, devices, mount points...
These are the domain of the service supervisor, e.g. systemd for NixOS, supervisord, or even simply a user running things manually.

It does not, but it knows what features are supported by process manager.

These options could be abstracted as follows: ...

That's up to process manager implementation that will be using sal. In docker, clearly services will not be abstracted like this. Still thanks for idea for later implementation.

The SAL services should be functions that take a configuration and return an attrset with e.g.:

Sal services are schema definitions and not a function, how you implement them is up to process managers.

Note that Type can only be "simple", "oneshot" or "forking" and the other types will need to be overridden when using systemd.

Thanks, i forgot for oneshot services, will add them.

NixOS's systemd will need to know about PermissionsStartOnly=true

I'm working on abstraction that will define for each phase what privileges are needed, so no direct definition of this (limited) systemd speciffic option. This was mostly for directory creation in systemd and i'm abstraction storage too.

I have to add things i'm working on:

• storage abstraction (with access control lists)
• sockets support
• process privileges and process pre/post phases privileges

[wmertens]

Do you mean "whether process manager can handle forking daemons."?

[offlinehacker]

Yes, i'm adding a quite more flags, for different features.
On Dec 16, 2014 5:42 PM, "wmertens" notifications@github.com wrote:

In services/services.nix
#5246 (diff):

+let
+in {

• options = {
• sal.services = mkOption {

•  default = {};
  

•  type = types.attrsOf types.optionSet;
  

•  options = [ serviceOptions ];
  

•  description = "Definition of systemd service units.";
  

• };

• sal.processManager = {

•  canFork = mkOption {
  

•    default = false;
  

•    type = types.bool;
  

•    description = "Whether process mananager can fork.";
  

Do you mean "whether process manager can handle forking daemons."?

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/pull/5246/files#r21911916.

[wmertens]

Systemd assumes to much and at the same time limits portability on anything but systemd. Docker has to deal with data volumes(where services store their data) and ports exposion betwene services, so let's think out of the systemd box(because it's really just a very limited box).

I was really just looking for a list of all things a supervisor could handle, looks like indeed systemd doesn't directly support those two use cases but neither does Docker really... either way glue code is needed no?

Multiple instances of the same service can exist, as long as they don't have conflicting resource use, e.g. socket binds, single-writer paths.

That's not something sal will deal with, sal is only about definitions. I think it's not too hard to add multy service support.

Hmmm... so you'd give the SAL functions a config with overridden config.service.x for each instance of a service? That might work...

The SAL does NOT do:
service supervision
logging stdout/stdenv
userids, permissions, limits, timers, devices, mount points...
These are the domain of the service supervisor, e.g. systemd for NixOS, supervisord, or even simply a user running things manually.

It does not, but it knows what features are supported by process manager.

I was afraid that would cause supervisor-specific code into the services. It would be nice to write a service definition and have it work on new supervisors without change. Unfounded fear?

These options could be abstracted as follows: ...

That's up to process manager implementation that will be using sal. In docker, clearly services will not be abstracted like this. Still thanks for idea for later implementation.

Yeah I was focusing too much on an abstracted interface at the filesystem level, but indeed that's not necessary because we control the supervisor as well as the service. So all you need is the service information and then the supervisor can implement it somehow.

I have to add things i'm working on:

• storage abstraction (with access control lists)
• sockets support
• process privileges and process pre/post phases privileges

Please push changes 😁.

So, how do you envision the workflow for a user that wants to run nginx as himself for development, or e.g. a CMS deployment via nixpkgs on an Ubuntu server?

[7c6f434c]

As I understood, there is a decision to make, whether SAL tries to express as little as definitely necessary or as much as can be specified in a portable way (so the service config generator can make its own decision about what information to use — without needing any more data in the simple cases).

[offlinehacker]

Fixed bugs with supervisord and docker. Assertions are now handled properly(and better) and not dependant on nixos assertions.

You can test now with going to services/process-managers/supervisord and running:

nix-build default.nix --argstr configuration $PWD/test.nix --show-trace --argstr name test

This will create configuration and supervisord wrapper scripts in bin directory:

test-start-services
test-control-services
test-stop-services

You can do the same with docker+fig by going to services/process-managers/docker and running:

nix-build fig.nix --argstr configuration $PWD/test.nix --show-trace --argstr name test

To run: fig -f result up

[offlinehacker]

Do you have some ideas how services with different process managers(nixos+systemd, supervisord, docker+fig...) could be tested on all of them(format of tests, debugging of tests,...) ? Testing is really critical feature that has to be in service abstraction layer.

[sofuture]

Hey @offlinehacker this is also something I'd really like to see. How can I help?

[copumpkin]

Any progress on this?

[garbas]

@offlinehacker is this something you're still working on, or this became stale?

[offlinehacker]

I'm closing this pull request, as after two years of dealing with docker, kubernetes and other containarized environments, i have much better idea how nix-services should be designed. Now all i need is a bit of time, which i would have to take, because it's something we need quite urgent on gatehub and proteuslabs. This work is from my perspecitve obsolete.

[copumpkin]

@offlinehacker if it's obsolete, do you have something better? Curious where we should take this going forward

[offlinehacker]

@copumpkin yes we should, I have some cool ideas, especially related with docker/kubernetes and other cluster technologies. I have built nix-kubernetes, but this is just abstraction over kubernetes. I want to make nix-services environment independant and cluster aware. The only problem is limited amount of time I have, if you have some time we can discuss how to push this forward.

[copumpkin]

Yeah, I'll check out your nix-kubernetes to see what you've done. It's mostly just been frustrating me to use dockerTools to build service containers knowing that we have lots of nicely modeled services sitting in NixOS, but we can't use any of them because they're expressed in a language that's too specific to systemd.

[domenkozar]

I hope you all saw https://github.com/LnL7/nix-darwin

[offlinehacker]

@domenkozar nice General idea is abstraction for different runtime environments, we could reuse module systems as backends for these environments.