-
-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC 0151] NixOS port allocator #151
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,135 @@ | ||||||||||
--- | ||||||||||
feature: nixos-port-alloc | ||||||||||
start-date: 2023-06-10 | ||||||||||
author: lucasew | ||||||||||
co-authors: (find a buddy later to help out with the RFC) | ||||||||||
shepherd-team: (names, to be nominated and accepted by RFC steering committee) | ||||||||||
shepherd-leader: (name to be appointed by RFC steering committee) | ||||||||||
related-issues: (will contain links to implementation PRs) | ||||||||||
--- | ||||||||||
|
||||||||||
# Summary | ||||||||||
[summary]: #summary | ||||||||||
|
||||||||||
A port allocator for NixOS services. | ||||||||||
|
||||||||||
# Motivation | ||||||||||
[motivation]: #motivation | ||||||||||
|
||||||||||
Sometimes people don't care about which port a service is running, only that it | ||||||||||
should be written somewhere so a service such as nginx can find it. | ||||||||||
|
||||||||||
# Detailed design | ||||||||||
[design]: #detailed-design | ||||||||||
|
||||||||||
The less likely used ports of the port space in a system are the higher ones, | ||||||||||
and the highest one is 65535, so a NixOS module could keep track of which services | ||||||||||
need a port and the service modules need only to reference that port in their | ||||||||||
configurations. | ||||||||||
Comment on lines
+25
to
+28
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The default range of ephemeral ports is 32768 to 60999, and using ports in this range for explicit binding can cause "funny" race conditions when an application There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe we can tie the default range to the sysctl setting for the ephemeral port range. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Found out that it's Cool, #TIL There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||||||||||
|
||||||||||
This module exposes the options under `networking.ports`. A service module can | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Considering the wording in here, this is supposed to be opt-in, correct? I.e. this won't be used (or enforced to be used) in other upstream NixOS modules itself, correct? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes. The allocator is opt-in. I specially use for services that are behind nginx so I don't need to think about which port is being used by each service (I got a few weird errors and conflicts before that and wanted to keep ports such as 3000 and 5000 free). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I wonder if it makes sense to use it by default, however in this case it would probably need to support a "preferred port". I'm thinking something like this:
So the port request options would be something like:
The downside would be that if you enable a new service it may shift an existing service which could cause issues if you were relying on the default. (For example you were running Mumble and connecting to mumble://yourhost.example without a port) This could cause runtime issues after switching to the new configuration. This is probably worse than an switch-time failure to start the new service. However for the reverse-proxy case it makes a lot of sense to have this behaviour as you can configure your reverse-proxy using something like In fact maybe it would make sense to use the port allocator for all services by default but use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If this should be actually used by default, another thing to check whether (and how much) this will increase the expensiveness of evaluation a NixOS system. |
||||||||||
request a port by defining `networking.ports.service.enable = true` and get the | ||||||||||
allocated port by referring to `networking.ports.service.port`. The service doesn't | ||||||||||
Comment on lines
+31
to
+32
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
On first read it wasn't clear to me that Alternative suggestions are |
||||||||||
depend on which logic the allocator uses to generate the port number. Only asks for | ||||||||||
a port and get the port to be used. | ||||||||||
|
||||||||||
The port allocator will allocate ports decremented from 65535 so it's very unlikely | ||||||||||
that it will reach ports under 1024. Because of the ordered nature of attrsets. | ||||||||||
|
||||||||||
|
||||||||||
# Examples and Interactions | ||||||||||
[examples-and-interactions]: #examples-and-interactions | ||||||||||
|
||||||||||
This is how the module would be used: | ||||||||||
```nix | ||||||||||
{ config, lib, ... }: | ||||||||||
lib.mkIf config.service.foo.enable { | ||||||||||
networking.ports.foo-web.enable = true; | ||||||||||
service.foo.port = mkDefault config.networking.ports.foo-web.port; | ||||||||||
} | ||||||||||
``` | ||||||||||
|
||||||||||
And an already working implementation of the specification: | ||||||||||
```nix | ||||||||||
{ config, lib, ... }: | ||||||||||
|
||||||||||
let | ||||||||||
inherit (builtins) removeAttrs; | ||||||||||
inherit (lib) mkOption types submodule literalExpression mdDoc mkDefault attrNames foldl' mapAttrs mkEnableOption attrValues; | ||||||||||
in | ||||||||||
|
||||||||||
{ | ||||||||||
options.networking.ports = mkOption { | ||||||||||
default = {}; | ||||||||||
|
||||||||||
example = literalExpression ''{ | ||||||||||
{ | ||||||||||
app.enable = true; | ||||||||||
} | ||||||||||
}''; | ||||||||||
|
||||||||||
description = "Build time port allocations for services that are only used internally"; | ||||||||||
|
||||||||||
apply = ports: lib.pipe ports [ | ||||||||||
(attrNames) # gets only the names of the ports | ||||||||||
(foldl' (x: y: x // { | ||||||||||
"${y}" = (ports.${y}) // { | ||||||||||
port = x._port; | ||||||||||
}; | ||||||||||
_port = x._port - 1; | ||||||||||
}) {_port = 65534; }) # gets the count down of the ports | ||||||||||
(x: removeAttrs x ["_port"]) # removes the utility _port entity | ||||||||||
]; | ||||||||||
|
||||||||||
type = types.attrsOf (types.submodule ({ name, config, options, ... }: { | ||||||||||
options = { | ||||||||||
enable = mkEnableOption "Enable automatic port allocation for service ${name}"; | ||||||||||
port = mkOption { | ||||||||||
description = "Allocated port for service ${name}"; | ||||||||||
type = types.nullOr types.port; | ||||||||||
}; | ||||||||||
}; | ||||||||||
})); | ||||||||||
}; | ||||||||||
|
||||||||||
config.environment.etc = lib.pipe config.networking.ports [ | ||||||||||
(attrNames) | ||||||||||
(foldl' (x: y: x // { | ||||||||||
"ports/${y}" = { | ||||||||||
inherit (config.networking.ports.${y}) enable; | ||||||||||
text = toString config.networking.ports.${y}.port; | ||||||||||
}; | ||||||||||
}) {}) | ||||||||||
]; | ||||||||||
} | ||||||||||
``` | ||||||||||
|
||||||||||
# Drawbacks | ||||||||||
[drawbacks]: #drawbacks | ||||||||||
|
||||||||||
- If someone externally expects to use that service directly, the port which could be used | ||||||||||
to access may differ like a local IP when it's not reserved by the router so it's not | ||||||||||
recommended to use this module in these cases. | ||||||||||
|
||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Another drawback of this approach is that port numbers are unstable, meaning ports of existing services will change whenever a new service's port is registered, leading to unnecessary restarts of completely unrelated services. I don't have a suggestion on how to address that, just something to note. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess it could be mostly mitigated by hashing the values within the available port range. Collisions are very likely in a range as small as this though, so we'd need to be pretty clever about handling those. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I know, I am having this problem lol There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Well, I think nix has a builtin to hash strings, maybe if I can get the first n letters then use a remainder operator... There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you wanna play around with it, here's a very naive way to map ports: let
pkgs = import <nixpkgs> { system = builtins.currentSystem; };
lib = pkgs.lib;
inherit (builtins) fromTOML fromJSON toJSON readFile hashString substring;
inherit (lib) mod;
inherit (lib.lists) foldl;
portFor = start: len: hash: start + (mod (fromTOML "x=0x${substring 0 8 hash}").x len);
portForLocalRange = portFor 32768 28230;
assignPort = ports: name: hash:
let
newHash = hashString "sha1" hash;
port = portForLocalRange newHash;
in
if ports?${toString port}
then assignPort ports name newHash
else ports // { ${toString port} = name; };
data = map (x: x.value) (fromJSON (readFile ./data.json));
in
toJSON (foldl (ports: name: assignPort ports name name) { } data) $ nix eval --raw -f ports.nix | jq
{
"32775": "odit-sunt-quidem-voluptatem-ut-harum-et-hic",
"32798": "voluptates-quis-corrupti-rerum-ullam-sunt",
"32803": "autem-ab-eum-error-tempore-ut",
"32807": "aspernatur-qui-omnis-omnis-temporibus-est-eum",
"32820": "doloremque-quidem-dolorem-occaecati-consequatur-consequatur-possimus-quia-voluptas",
...
"60937": "sed-vero-est-dolores-aspernatur",
"60960": "ducimus-voluptas-vel-libero-ut-tempore-nobis-magni-labore",
"60975": "aspernatur-ipsam-earum-beatae-quibusdam",
"60982": "aut-delectus-totam-labore",
"60989": "ipsum-voluptas-omnis-magnam-officiis-nam"
} There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yep. Still, if the port registration was lazy (only registered when you actually enable a service), the number of running services would be small in practice so collisions would be very unlikely and they could be handled with some form of backup strategy. This still feels like a weird assumption for an official module though. Using state is probably more reasonable, though you'd preferably want it at eval-time so you can actually reference the ports without having to resort to runtime wrappers. Not sure how you could do this without resorting to I guess if you manage the state file manually (i.e., just append names to it), you technically have automatic port allocation. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Well, in case of a collision, it's possible to implement a fallback strategy that sums 1 to one of the conflicting ports until there is no conflicts. Seems like a good balance between not triggering a restart and avoiding conflicts. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
It's worth noting that you can have mostly stable port numbers with consistent hashing and some simple collision handling. Yes, if a collision occurs at least one port will change but the average server won't have too many services so collisions will probably be somewhat rare. I also think it is important to have the ports available at eval time. It is frequently important to configure a reverse proxy, firewall or other config files. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For collision handling, hash tables are relevant, and open addressing in particular: https://en.wikipedia.org/wiki/Hash_table#Open_addressing You could consider a concept of "priority", which, translated to hash table language, means that entries with high priorities are inserted first, so they get the best spots and are most stable: it takes an entry with a higher priority to cause it to move ports (existence of a colliding entry/entries with an equal or higher priority in either the old or new configuration). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure that we really want priority. It seems overly complicated and error prone. I was thinking sort of two levels of priority. "Required" and "Random". "Required" gives an error on conflict with another "Required" and "Random" is just consistently hashed. I considered "Preferred" but this just seems like it will lead to people assuming that they get the preferred port then causing issues if it gets reassigned, if it is free to be shuffled I think it is best to never pretend to have some sort of stable preferred value. I wrote more about this here: #151 (comment) |
||||||||||
# Alternatives | ||||||||||
[alternatives]: #alternatives | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To sum up some of the other threads into an alternative proposal/adjustments to the original proposal. Different ModesThe current proposal only deals with automatically assigned ports. This means that it is unaware of static ports on the system. I think it may be better to make the port allocator (maybe better called a "manager" now) know about all ports to make its decision. I would add two or three modes to the allocator
Most notably Mostly Stable AssignmentI would update the assignment algorithm to be mostly stable. This reduces disruption when changing enabled services. In the vast majority of changes there would be no reassignment. In cases where there would have been a collision only a small number of services (hopefully 1) would change ports. The most likely method of doing this would be consistent hashing using the port name. On collision a new hash would be run to pick an alternate port. Reserved RangesI think the allowed or reserved ranges should be customizable. By default we should probably exclude 0-1024 (root-only) and the ephemeral port range. Different IPsIt would also be nice to be able to reserve ports separately on different IPs. This is probably a nice-to-have and can be implemented later. If I reserve some ports on This would be forwards-compatible to add later. Ports that don't specify an IP are assumed to use networking.ports.foo = {
address = "127.0.0.8";
}; SchemaMy current thought is the flowing schema:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should maybe mention socket activation. The canonical solution to running services behind nginx, without having to worry about allocating unique ports to them, is to have the service listen on a unix socket, and point nginx to that. (This also makes access controls easier.) This isn't supported by every service, which is where the value of this RFC comes in, but it's probably a better solution when it is supported. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: Socket activation is orthogonal to UNIX sockets. You can also do socket activation for IP-based sockets. I agree that it is worth mentioning UNIX sockets as an alternative ("infinite namespace" prevents collisions, file permissions are useful for security and the performance is better) but I think the downsides of "not all services support UNIX sockets" and "doesn't support remote access" is enough justification for this RFC. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit² : can't you mount a UNIX socket over the network? :-) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You can? I've only heard of using something like netcat to proxy a TCP socket into a UNIX socket. Does NFS or something support that? How does file descriptor transfer work? (or does it just fail?) ...but I think we are getting off-topic now. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I wouldn't quite call it orthogonal. Theoretically, yes, but in practice, it tends to be the only way to have an application designed for TCP to use a Unix socket.
Yes, that's why I said it should be mentioned as an alternative considered rather than saying I opposed the RFC! |
||||||||||
|
||||||||||
Keep track of which ports have been used by services and often just seeing that | ||||||||||
the port is already being used by some other service when the activation logs show | ||||||||||
that the service failed to start. | ||||||||||
|
||||||||||
Forbid usage of common utility ports like 8080, 8081, 5000, 3000 and 3333. | ||||||||||
|
||||||||||
# Unresolved questions | ||||||||||
[unresolved]: #unresolved-questions | ||||||||||
|
||||||||||
How to allocate blocks of ports so something like a torrent client can use that to | ||||||||||
listen for p2p traffic? | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You could have a "hash function" that takes either a name or a port. If it's a port, it just returns the port. |
||||||||||
|
||||||||||
How to reserve higher ports to services so the automatic allocator skips them? | ||||||||||
|
||||||||||
# Future work | ||||||||||
[future]: #future-work | ||||||||||
|
||||||||||
Selfhosted toolkits that configure services behind a reverse proxy like nginx that | ||||||||||
doesn't need to care which local port services are listening to. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't that something you can achieve by using
config.services.yourservice.settings.port
everywhere (or something similar)? Port collisions of service-ports (i.e. 3000/8000/etc) are - in my experience at least - relatively rare, at least I can't actually recall a single time where I got bitten by this.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. Now you can set ports but sometimes you don't care which port as long as the two ends are using the same one.