Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0151] NixOS port allocator #151

Closed
wants to merge 3 commits into from
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 135 additions & 0 deletions rfcs/0151-nixos-port-alloc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
---
feature: nixos-port-alloc
start-date: 2023-06-10
author: lucasew
co-authors: (find a buddy later to help out with the RFC)
shepherd-team: (names, to be nominated and accepted by RFC steering committee)
shepherd-leader: (name to be appointed by RFC steering committee)
related-issues: (will contain links to implementation PRs)
---

# Summary
[summary]: #summary

A port allocator for NixOS services.

# Motivation
[motivation]: #motivation

Sometimes people don't care about which port a service is running, only that it
should be written somewhere so a service such as nginx can find it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that something you can achieve by using config.services.yourservice.settings.port everywhere (or something similar)? Port collisions of service-ports (i.e. 3000/8000/etc) are - in my experience at least - relatively rare, at least I can't actually recall a single time where I got bitten by this.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Now you can set ports but sometimes you don't care which port as long as the two ends are using the same one.


# Detailed design
[design]: #detailed-design

The less likely used ports of the port space in a system are the higher ones,
and the highest one is 65535, so a NixOS module could keep track of which services
need a port and the service modules need only to reference that port in their
configurations.
Comment on lines +25 to +28
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default range of ephemeral ports is 32768 to 60999, and using ports in this range for explicit binding can cause "funny" race conditions when an application connecting to a network port happens to occupy the chosen port. Using ports below 32768 is probably better for this reason.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can tie the default range to the sysctl setting for the ephemeral port range.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found out that it's net.ipv4.ip_local_port_range

Cool, #TIL

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not nix defined by default so the implementation will need a fallback

Default value is

sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768    60999

Screenshot_20230616-191031


This module exposes the options under `networking.ports`. A service module can
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering the wording in here, this is supposed to be opt-in, correct? I.e. this won't be used (or enforced to be used) in other upstream NixOS modules itself, correct?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The allocator is opt-in. I specially use for services that are behind nginx so I don't need to think about which port is being used by each service (I got a few weird errors and conflicts before that and wanted to keep ports such as 3000 and 5000 free).

Copy link
Contributor

@kevincox kevincox Jun 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if it makes sense to use it by default, however in this case it would probably need to support a "preferred port". I'm thinking something like this:

  1. Services set a "preferred port" and register with the allocator.
  2. Upon collision all but one of the colliding services are reassigned.
  3. There is an option to force a specific port for a service (and get an eval error if there is a conflict).

So the port request options would be something like:

  • Prefer(port)
  • Require(port)
  • NoPreference

The downside would be that if you enable a new service it may shift an existing service which could cause issues if you were relying on the default. (For example you were running Mumble and connecting to mumble://yourhost.example without a port) This could cause runtime issues after switching to the new configuration. This is probably worse than an switch-time failure to start the new service. However for the reverse-proxy case it makes a lot of sense to have this behaviour as you can configure your reverse-proxy using something like config.service.foo.port and it would auto-update.

In fact maybe it would make sense to use the port allocator for all services by default but use Require(port). That way 1. The port allocator won't assign that port to another service and 2. You will get an eval-time error if you enable two services that use the same port. Then the user could configure an override with a different port or with NoPreference. New services that bind to an internal IP could maybe considering using NoPreference by default as it won't cause any backwards incompatibility.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this should be actually used by default, another thing to check whether (and how much) this will increase the expensiveness of evaluation a NixOS system.

request a port by defining `networking.ports.service.enable = true` and get the
allocated port by referring to `networking.ports.service.port`. The service doesn't
Comment on lines +31 to +32
Copy link
Contributor

@kevincox kevincox Jun 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
request a port by defining `networking.ports.service.enable = true` and get the
allocated port by referring to `networking.ports.service.port`. The service doesn't
request a port by defining `networking.ports.${service}.enable = true` and get the
allocated port by referring to `networking.ports.${service}.port`. The service doesn't

On first read it wasn't clear to me that service was a placeholder. It looks similar to services.foo or systemd.services.foo where it is a literal. I think it would be good to make it more clear.

Alternative suggestions are networking.ports.foo, networking.ports.someservice , networking.ports.${service-name}, networking.ports.*, networking.ports.SERVICE. I think either of these would be more obviously a placeholder.

depend on which logic the allocator uses to generate the port number. Only asks for
a port and get the port to be used.

The port allocator will allocate ports decremented from 65535 so it's very unlikely
that it will reach ports under 1024. Because of the ordered nature of attrsets.


# Examples and Interactions
[examples-and-interactions]: #examples-and-interactions

This is how the module would be used:
```nix
{ config, lib, ... }:
lib.mkIf config.service.foo.enable {
networking.ports.foo-web.enable = true;
service.foo.port = mkDefault config.networking.ports.foo-web.port;
}
```

And an already working implementation of the specification:
```nix
{ config, lib, ... }:

let
inherit (builtins) removeAttrs;
inherit (lib) mkOption types submodule literalExpression mdDoc mkDefault attrNames foldl' mapAttrs mkEnableOption attrValues;
in

{
options.networking.ports = mkOption {
default = {};

example = literalExpression ''{
{
app.enable = true;
}
}'';

description = "Build time port allocations for services that are only used internally";

apply = ports: lib.pipe ports [
(attrNames) # gets only the names of the ports
(foldl' (x: y: x // {
"${y}" = (ports.${y}) // {
port = x._port;
};
_port = x._port - 1;
}) {_port = 65534; }) # gets the count down of the ports
(x: removeAttrs x ["_port"]) # removes the utility _port entity
];

type = types.attrsOf (types.submodule ({ name, config, options, ... }: {
options = {
enable = mkEnableOption "Enable automatic port allocation for service ${name}";
port = mkOption {
description = "Allocated port for service ${name}";
type = types.nullOr types.port;
};
};
}));
};

config.environment.etc = lib.pipe config.networking.ports [
(attrNames)
(foldl' (x: y: x // {
"ports/${y}" = {
inherit (config.networking.ports.${y}) enable;
text = toString config.networking.ports.${y}.port;
};
}) {})
];
}
```

# Drawbacks
[drawbacks]: #drawbacks

- If someone externally expects to use that service directly, the port which could be used
to access may differ like a local IP when it's not reserved by the router so it's not
recommended to use this module in these cases.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another drawback of this approach is that port numbers are unstable, meaning ports of existing services will change whenever a new service's port is registered, leading to unnecessary restarts of completely unrelated services. I don't have a suggestion on how to address that, just something to note.

Copy link
Member

@szlend szlend Jun 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it could be mostly mitigated by hashing the values within the available port range. Collisions are very likely in a range as small as this though, so we'd need to be pretty clever about handling those.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another drawback of this approach is that port numbers are unstable, meaning ports of existing services will change whenever a new service's port is registered, leading to unnecessary restarts of completely unrelated services. I don't have a suggestion on how to address that, just something to note.

I know, I am having this problem lol

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it could be mostly mitigated by hashing the values within the available port range. Collisions are very likely in a range as small as this though, so we'd need to be pretty clever about handling those.

Well, I think nix has a builtin to hash strings, maybe if I can get the first n letters then use a remainder operator...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you wanna play around with it, here's a very naive way to map ports:

let
  pkgs = import <nixpkgs> { system = builtins.currentSystem; };
  lib = pkgs.lib;

  inherit (builtins) fromTOML fromJSON toJSON readFile hashString substring;
  inherit (lib) mod;
  inherit (lib.lists) foldl;

  portFor = start: len: hash: start + (mod (fromTOML "x=0x${substring 0 8 hash}").x len);
  portForLocalRange = portFor 32768 28230;

  assignPort = ports: name: hash:
    let
      newHash = hashString "sha1" hash;
      port = portForLocalRange newHash;
    in
    if ports?${toString port}
    then assignPort ports name newHash
    else ports // { ${toString port} = name; };

  data = map (x: x.value) (fromJSON (readFile ./data.json));
in
toJSON (foldl (ports: name: assignPort ports name name) { } data)
$ nix eval --raw -f ports.nix | jq
{
  "32775": "odit-sunt-quidem-voluptatem-ut-harum-et-hic",
  "32798": "voluptates-quis-corrupti-rerum-ullam-sunt",
  "32803": "autem-ab-eum-error-tempore-ut",
  "32807": "aspernatur-qui-omnis-omnis-temporibus-est-eum",
  "32820": "doloremque-quidem-dolorem-occaecati-consequatur-consequatur-possimus-quia-voluptas",
  ...
  "60937": "sed-vero-est-dolores-aspernatur",
  "60960": "ducimus-voluptas-vel-libero-ut-tempore-nobis-magni-labore",
  "60975": "aspernatur-ipsam-earum-beatae-quibusdam",
  "60982": "aut-delectus-totam-labore",
  "60989": "ipsum-voluptas-omnis-magnam-officiis-nam"
}

Copy link
Member

@szlend szlend Jun 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can't avoid state while also having stable port numbers. Even if you try to make it stateless with hashing, you'll run into problems when you have collisions.

Yep. Still, if the port registration was lazy (only registered when you actually enable a service), the number of running services would be small in practice so collisions would be very unlikely and they could be handled with some form of backup strategy. This still feels like a weird assumption for an official module though.

Using state is probably more reasonable, though you'd preferably want it at eval-time so you can actually reference the ports without having to resort to runtime wrappers. Not sure how you could do this without resorting to --impure or sandbox-paths though.

I guess if you manage the state file manually (i.e., just append names to it), you technically have automatic port allocation.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, in case of a collision, it's possible to implement a fallback strategy that sums 1 to one of the conflicting ports until there is no conflicts. Seems like a good balance between not triggering a restart and avoiding conflicts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can't avoid state while also having stable port numbers.

It's worth noting that you can have mostly stable port numbers with consistent hashing and some simple collision handling. Yes, if a collision occurs at least one port will change but the average server won't have too many services so collisions will probably be somewhat rare.

I also think it is important to have the ports available at eval time. It is frequently important to configure a reverse proxy, firewall or other config files.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For collision handling, hash tables are relevant, and open addressing in particular: https://en.wikipedia.org/wiki/Hash_table#Open_addressing

You could consider a concept of "priority", which, translated to hash table language, means that entries with high priorities are inserted first, so they get the best spots and are most stable: it takes an entry with a higher priority to cause it to move ports (existence of a colliding entry/entries with an equal or higher priority in either the old or new configuration).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that we really want priority. It seems overly complicated and error prone. I was thinking sort of two levels of priority. "Required" and "Random". "Required" gives an error on conflict with another "Required" and "Random" is just consistently hashed. I considered "Preferred" but this just seems like it will lead to people assuming that they get the preferred port then causing issues if it gets reassigned, if it is free to be shuffled I think it is best to never pretend to have some sort of stable preferred value.

I wrote more about this here: #151 (comment)

# Alternatives
[alternatives]: #alternatives
Copy link
Contributor

@kevincox kevincox Jun 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To sum up some of the other threads into an alternative proposal/adjustments to the original proposal.

Different Modes

The current proposal only deals with automatically assigned ports. This means that it is unaware of static ports on the system. I think it may be better to make the port allocator (maybe better called a "manager" now) know about all ports to make its decision.

I would add two or three modes to the allocator

  1. { port = 1234 }. This is required mode In this mode the specific port is reserved for this service. It will not be assigned to another service. An error is raised at eval time if two services require the same port.
  2. {} This is random mode In this mode an otherwise non-allocated port is assigned.
  3. { mode = "prefer"; port = 1234 } The port will be assigned to one of the preferred mode services if not taken by a required mode service. If the port is taken by a required mode or another preferred mode service it is given a port as if it was random mode. (IDK if this is necessary, maybe we can skip it for the first pass. The main benefit is aesthetics but it raises the chance of a re-assignment causing a problem if people are relying on the preferred port. It can always be added later.)

Most notably mode = "require" allows existing modules to use the port allocator and remain backwards compatible. This provides good error messages on conflict and avoids collisions between these services and randomly assigned ports. I would recommend migrating all existing services to mode = "require".

Mostly Stable Assignment

I would update the assignment algorithm to be mostly stable. This reduces disruption when changing enabled services. In the vast majority of changes there would be no reassignment. In cases where there would have been a collision only a small number of services (hopefully 1) would change ports.

The most likely method of doing this would be consistent hashing using the port name. On collision a new hash would be run to pick an alternate port.

Reserved Ranges

I think the allowed or reserved ranges should be customizable. By default we should probably exclude 0-1024 (root-only) and the ephemeral port range.

Different IPs

It would also be nice to be able to reserve ports separately on different IPs. This is probably a nice-to-have and can be implemented later. If I reserve some ports on 127.0.0.1 it would be nice if I can also use those on my public IP. Maybe we could even assign IP addresses out of a range like 127.0.0.0/8. But this is probably fairly niche. In most cases assigning ports globally is unlikely to run out and is likely simpler when trying to debug issues on a system as there should be no port reuse so you can mostly ignore IPs.

This would be forwards-compatible to add later. Ports that don't specify an IP are assumed to use :: (and collide with all ports on all addresses).

networking.ports.foo = {
  address = "127.0.0.8";
};

Schema

My current thought is the flowing schema:

networking.ports.assignmentRange.start

The first port number that is eligible for assignment. Defaults to 1.

networking.ports.assignmentRange.end

The first port number that is not eligible for assignment. Defaults to 2^16.

networking.ports.reservedRanges

List of {start = 0; end = 1024} ranges that won't be assigned from. Defaults to [{start = 0; end = 1024} {start = ephemeral_start; end =- ephemeral_end}

networking.ports.port.*

Attrset of ports. Presence indicates that a port should be allocated. (I can't think of a use case to disable, it would likely cause an eval failure for whatever reads the port). This could likely be proxied via services.*.port or similar in many cases.

networking.ports.port.*.address

(Possible future addition). Defaults to ::.

networking.ports.port.*.mode

(Possible future addition). Used if port is set. Enum of "required" or "preferred". Defaults to "required".

networking.ports.port.*.port

Defaults to null. If set this port is fixed and used only for avoidance for other assignments and raising an error on conflict between manually-assigned ports. This is also set by the allocator so that it can be referenced as config.networking.ports.port.foo.port when generating config files for dependent services.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should maybe mention socket activation. The canonical solution to running services behind nginx, without having to worry about allocating unique ports to them, is to have the service listen on a unix socket, and point nginx to that. (This also makes access controls easier.) This isn't supported by every service, which is where the value of this RFC comes in, but it's probably a better solution when it is supported.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Socket activation is orthogonal to UNIX sockets. You can also do socket activation for IP-based sockets.

I agree that it is worth mentioning UNIX sockets as an alternative ("infinite namespace" prevents collisions, file permissions are useful for security and the performance is better) but I think the downsides of "not all services support UNIX sockets" and "doesn't support remote access" is enough justification for this RFC.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit² : can't you mount a UNIX socket over the network? :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can? I've only heard of using something like netcat to proxy a TCP socket into a UNIX socket. Does NFS or something support that? How does file descriptor transfer work? (or does it just fail?)

...but I think we are getting off-topic now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Socket activation is orthogonal to UNIX sockets. You can also do socket activation for IP-based sockets.

I agree that it is worth mentioning UNIX sockets as an alternative ("infinite namespace" prevents collisions, file permissions are useful for security and the performance is better) but I think the downsides of "not all services support UNIX sockets" and "doesn't support remote access" is enough justification for this RFC.

I wouldn't quite call it orthogonal. Theoretically, yes, but in practice, it tends to be the only way to have an application designed for TCP to use a Unix socket.

I think the downsides of "not all services support UNIX sockets" and "doesn't support remote access" is enough justification for this RFC.

Yes, that's why I said it should be mentioned as an alternative considered rather than saying I opposed the RFC!


Keep track of which ports have been used by services and often just seeing that
the port is already being used by some other service when the activation logs show
that the service failed to start.

Forbid usage of common utility ports like 8080, 8081, 5000, 3000 and 3333.

# Unresolved questions
[unresolved]: #unresolved-questions

How to allocate blocks of ports so something like a torrent client can use that to
listen for p2p traffic?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could have a "hash function" that takes either a name or a port. If it's a port, it just returns the port.
Then use a high priority, and check that it was possible to allocate them at the desired individual port numbers, throwing an error if a collision happened anyway. (ie "required" mode mentioned before)


How to reserve higher ports to services so the automatic allocator skips them?

# Future work
[future]: #future-work

Selfhosted toolkits that configure services behind a reverse proxy like nginx that
doesn't need to care which local port services are listening to.