Develop a universal web service definition

[mbrgm]

NixOS needs a universal, webserver-agnostic webservice definition. Currently, most webservices are implemented as subservices for Apache httpd, but users should be free to choose the webserver they want to use. The wish for this definition was already mentioned in several issues, but afaik (please correct me if I'm wrong) not worked on in a coordinated way. To start, let's try to compile a list of actionable steps. Please feel free to contribute to this issue, add ToDo's to the list, offer your opinion on several topics or explain how you would propose to solve this problem.

ToDo

• Compile an overview of the status quo
  • Reference existing related issues
  • Create a list of existing httpd subservices
  • Create a list of other webservices, which are not implemented httpd subservices
• Develop a concept for a universal web service definition
  • Collect what should be included, what should be left out
  • Discuss how a web service definition could be implemented (regular services.<name>, in the style of httpd subservices or something different...)
  • Define the interface for a web service definition
  • Implement the interface
• Test the new web service definition
  • Investigate the current implementation of web services testing
  • Implement testing for the web service definition (test per webserver?)
• Move existing webservices to new web service definition
  • Create new web service definition for existing httpd subservices
  • Create web service definition for web services, which are not implemented as a httpd subservice
  • Deprecate old httpd subservices?
  • Deprecate old non-httpd web services?

Related issues

• services.httpd: Improve virtualHosts and subservices. #6960 mentions having worked on a system, which is more intuitive than the current subservices system
• Rethink httpd subservices #18977 proposes the definition of webservices as a regular NixOS service with individual webserver configs being an attribute of that service.
• nixos/lighttpd: add Nextcloud sub-service #19703 wants to implement a NextCloud subservice for lighttpd; the need for a universal web service definition was also mentioned

[jluttine]

This is a great suggestion. I would really like to see this moving forward. I'm new to Nix(OS) but as there hasn't been any activity for some time, I thought I'd sketch some ideas here just in case they are worth anything.

First, I like the syntax used to define sub-services for Apache. I just switched subServices to webApps, so the syntax could look something like (just rough sketching to get an idea):

services.nginx.virtualHosts."mydomain.com" = {
  ...
  webApps = [
    {
      app = "nextcloud";
      baseURL = "nextcloud/";
      database = "mysql";
      dbName = "nextcloud";
      ...
    }
    {
      app = "wordpress";
      baseURL = "wordpress/";
      database = "postgresql";
      dbName = "wordpress";
      ...
    }
    ...
  ];
  ...
};

The attribute app is used to determine the web application. The web application is given the config set from which the attribute app has been removed and some other attributes are added in order to tell the web application some server configuration, for instance, web server is nginx and virtual host is mydomain.com. Thus, each HTTP server service parses the lists of web apps for each virtual host and configures the web apps. The web app services are defined elsewhere (for instance, services.webapps.<appname>. That is, for each entry in the webApps list (lets call a single set entry appCfg), the web server service writes something like:

services.webapps."${appCfg.app}" = [
  (removeAttrs appCfg ["app"]) // {
    httpd = "nginx";
    vhost = "mydomain.com";
  };
];

Each services.webapps.<appname> is a list of configurations. This allows one to set up multiple instances of each web app, possibly even using different HTTP servers for them. Because each web app is given the HTTP server type (nginx, apache, lighttpd), they can make conditional configurations.

If I understood the current Apache sub-services solution correctly, the sub-services could only configure the virtual host (or the main server) but not other services. For instance, the sub-services couldn't configure databases, php-fpm or uWSGI instances. I might be wrong here. But anyway, configuring those should be possible and this approach should make it easily possible (I think).

So the main points in my suggestion are:

• User configures web app instances under the virtual host configuration using subServices or webApps or similar attribute.

• Each web app can have multiple instances.

• Each web app should get the information about which HTTP server is used so they can make nginx/apache/lighttpd specific customizations.

• Web apps can make "global" configurations, for instance, they can configure php-fpm or uWSGI services.

[ericsagnes]

Ideas expressed in #26067, a generic approach to service abstraction, could be a source of inspiration to abstract web services too.

[mmahut]

Are there any updates on this issue, please?

[1000101]

Has there been any progress, please? I would like to modify #77830 so that the user could decide which web server to use but I'm not really sure if there's a nice way of implementing it.

[stale]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[florianjacob]

There has been no progress with this issue as far as I know, however even if a universal web service definition is hard, some convention on how to do it (e.g. a service.name.enable-nginx-vhost = true; would help.

[Melkor333]

There is the nixcloud repository which may be very related to this issue. The Nixcloud project aims to deliver services in a very unified way behind a nginx reverse proxy which handles ssl, domain mapping. The configuration of services behind the proxy have been unified to allow for a rather simple configuration of services. There's also other stuff like helper variables for DB/DB-user creation on first execution, etc.

Maybe some principles from this project might be integrated into Nixpkgs... I don't know exactly what a "universal web services definition" means to you but in my experience with Debian I liked to use an Nginx proxy and run web services with the developer given webserver config (which was often apache or some language specific minimal webserver). This abstraction let me handle the incoming connections "my way" while not having to translate configurations from other webservices.
I don't know if there are serious negative side effects of using a reverse proxy (e.g. higher resource usage) but if that's not the case then I would be in favour of something like this, as the nixcloud project is a great example showing that this approach can work. And even if you think that this idea is too "forceful" for nixpkgs I highly encourage you to check the project out! 😃

[stale]

I marked this as stale due to inactivity. → More info

[mohe2015]

definitely not stale

[makefu]

any update on that? i'd love to see this in nixpkgs :)

[aanderse]

There is no work on a generic "fits all" solution here that I am aware of. Some people have been working on making a few NixOS modules support multiple web server backends, but in a very manual way that is adds a relatively large amount of code to maintain per module.

[makefu]

tbh i was thinking the same, still it would be pretty cool :)
I am asking because the mediawiki service seems to currently only support apache and this issue would solve the situation.

[aanderse]

Yes. Inspiration and PRs welcome 😆