Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add network library #299409

Open
wants to merge 28 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
ad9df7e
Started the design document for the network library.
djacu Mar 26, 2024
79dc1bc
Updated network library using official nixfmt.
djacu Mar 26, 2024
33747ae
Added network tests into the library tests.
djacu Mar 26, 2024
5730535
Swapped the subnet mask function. Added tests for the new encode func…
djacu Mar 26, 2024
4cb8f2d
Added more tests
djacu Mar 26, 2024
5a963ec
Document and test the pow function.
djacu Mar 26, 2024
51de0d0
Added more logic for the ipv4 encode function. Added docs and more
djacu Mar 26, 2024
ee3d0f8
Documented the function that verifies valid prefix length and added t…
djacu Mar 27, 2024
de10daa
Reworked the logic of the function that verifies the prefix length fr…
djacu Mar 27, 2024
5898ff8
Added documentation for the function that converts a prefix length to…
djacu Mar 27, 2024
f5ab94e
Added documentation for the function that makes IPv4Address attrsets …
djacu Mar 27, 2024
52eef00
Added more to the readme.
djacu Mar 27, 2024
0e3651b
Changed function name that ingests a cidr and converts it to an IPv4A…
djacu Mar 27, 2024
ab3707a
Revert test-with-nix.nix back to original state.
djacu Apr 2, 2024
e1f9b01
Added network tests back in test-with-nix.nix.
djacu Apr 2, 2024
7e55177
Added entry in doc/default.nix.
djacu Apr 2, 2024
5ea3344
Changed _makeIPv4 to _parse to be more descriptive and consistent.
djacu Apr 2, 2024
6116326
Added a function to verify ipv4 addresses and more tests.
djacu Apr 2, 2024
3a09ebe
Switch to foldl' per review recommendation.
djacu Apr 11, 2024
18e64c7
Update lib/network/internal.nix
djacu Apr 11, 2024
24c77da
Move the documentation of the IPv4Address attrset from the readme to …
djacu Apr 12, 2024
f28ca5a
Add doc comment for ipv4.fromCidrString
djacu Apr 12, 2024
b7c93e1
Remove subnet mask from the library for the initial PR.
djacu Apr 12, 2024
a8ab014
Removing _encode internal function as it was only used by subnet mask…
djacu Apr 12, 2024
61b60a3
Clean up tests to limit the references to internal functions.
djacu Apr 12, 2024
ff6f0e2
Fix working in tests.
djacu Apr 12, 2024
c27b3d6
inherit cleanup since subnet mask was removed.
djacu Apr 12, 2024
b871844
Fix network api so the manual renders.
djacu Apr 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ let
{ name = "path"; description = "path functions"; }
{ name = "filesystem"; description = "filesystem functions"; }
{ name = "fileset"; description = "file set functions"; }
{ name = "network"; description = "networking functions"; }
{ name = "sources"; description = "source filtering functions"; }
{ name = "cli"; description = "command-line serialization functions"; }
{ name = "gvariant"; description = "GVariant formatted string serialization functions"; }
Expand Down
3 changes: 3 additions & 0 deletions lib/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,9 @@ let
# linux kernel configuration
kernel = callLibs ./kernel.nix;

# network
network = callLibs ./network;
djacu marked this conversation as resolved.
Show resolved Hide resolved

inherit (builtins) add addErrorContext attrNames concatLists
deepSeq elem elemAt filter genericClosure genList getAttr
hasAttr head isAttrs isBool isInt isList isPath isString length
Expand Down
22 changes: 22 additions & 0 deletions lib/network/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Network library

This is the internal contributor documentation.

## Goals

The main goal of the network library is to provide utility functions for IPv4 and IPv6 conversions.
It should have the following properties.

- Easy: The functions should have obvious semantics.
- Safe: Throw early and helpful errors when mistakes are detected.
- Lazy: Only compute values when necessary.

## Tests

Tests are declared in `tests.sh`.

## Other implementations and references

- [Haskell](https://hackage.haskell.org/package/ip)
- [Python](https://docs.python.org/3/library/ipaddress.html)
- [Rust](https://doc.rust-lang.org/std/net/enum.IpAddr.html)
60 changes: 60 additions & 0 deletions lib/network/default.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/*
<!-- This anchor is here for backwards compatibity -->
[]{#sec-network}

The `lib.network` library allows you to work with IPv4 addresses in CIDR notation.

## IPv4 {#sec-network-ipv4}

### Structure {#sec-network-ipv4-structure}

The `lib.network.ipv4` library provides ingestion functions the create an `IPv4Address` object.
This is an attribute set with these values:

- `cidr`: A CIDR.
- `address`: An IP address.
- `prefixLength`: A prefix length.
- `subnetMask`: A subnet mask.

- [`lib.network.ipv4.fromCidrString`](#function-library-lib.network.ipv4.fromCidrString):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not really familiar with the documentation syntax, but it looks like L19-21 content redundant with documentation of function below... Does it really need to be duplicated?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this comment even get rendered?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think fromCidrString was not meant to be part of the above list.


Creates an `IPv4Address` object from an IPv4 address in CIDR notation as a string.
*/

{ lib }:
let
internal = import ./internal.nix { inherit lib; };
in
{
/**
Creates an `IPv4Address` object from an IPv4 address in CIDR notation as a string.

# Example

```nix
fromCidrString "192.168.0.1/24"
=> {
address = "192.168.0.1";
cidr = "192.168.0.1/24";
prefixLength = "24";
}
```

# Type

```
fromCidrString :: String -> IPv4Address
```

# Arguments

- [cidr] An IPv4 address in CIDR notation.
*/
ipv4.fromCidrString =
cidr:
let
address = internal.ipv4._verifyAddress cidr;
prefixLength = internal.ipv4._verifyPrefixLength cidr;
in
internal.ipv4._parse address prefixLength;
}
149 changes: 149 additions & 0 deletions lib/network/internal.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
{ lib }:
let
inherit (lib)
concatStringsSep
elemAt
range
toInt
;

inherit (builtins) all any foldl';
in
{
common = {
/**
Given a base and exponent, calculates base raised to the exponent.

# Example

```nix
pow 2 3
=> 8
```
djacu marked this conversation as resolved.
Show resolved Hide resolved

# Type

```
pow :: Int -> Int -> Int
```

# Arguments

- [base] The base.
- [exponent] The exponent.

# Throws

- If the exponent is less than 0.
*/
pow =
base: exponent:
if exponent < 0 then
throw "lib.network.pow: Exponent cannot be negative."
else if exponent == 0 then
1
else
foldl' (acc: _: acc * base) 1 (range 1 exponent);
};

ipv4 = {
/**
Extracts a prefix length from a CIDR and verifies it is valid.

If an IP address is given without a CIDR, then a prefix length of 32 is returned.

# Example

```nix
_verifyPrefixLength "192.168.0.1/24"
=> 24
_verifyPrefixLength "192.168.0.1"
=> 32
```

# Type

```
_verifyPrefixLength :: String => String
```

# Arguments

- [cidr] An IPv4 CIDR.

# Throws

- If a CIDR was given with a slash but no prefix length following.
- If there were multiple slashes in the CIDR.
*/
_verifyPrefixLength =
cidr:
let
splitCidr = lib.splitString "/" cidr;
in
if (builtins.length splitCidr) == 1 then
"32"
else if (builtins.length splitCidr) > 2 then
throw "lib.network.ipv4: Could not verify prefix length for CIDR ${cidr}."
else
let
afterSlash = elemAt splitCidr 1;
in
if afterSlash == "" then
throw "lib.network.ipv4: CIDR ${cidr} has no prefix length."
else if toInt afterSlash > 32 || toInt afterSlash < 0 then
throw "lib.network.ipv4: CIDR ${cidr} has an out of bounds prefix length, ${afterSlash}."
else
afterSlash;

_verifyAddress =
cidr:
let
splitCidr = lib.splitString "/" cidr;
address = elemAt splitCidr 0;
splitAddress = lib.splitString "." address;
intOctets = map toInt splitAddress;
in
if (builtins.length splitAddress) != 4 then
throw "lib.network.ipv4: CIDR ${cidr} is not of the correct form."
else if any (x: x == "") splitAddress then
throw "lib.network.ipv4: CIDR ${cidr} has an empty octet."
else if !(all (x: x >= 0 && x <= 255)) intOctets then
throw "lib.network.ipv4: CIDR ${cidr} has an out of bounds octet."
else
address;

/**
Given an IP address and prefix length, creates an attribute set of network parameters.

# Example

```nix
_parse "192.168.0.1/24"
=> {
address = "192.168.0.1";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this return something like [ 192 168 0 1 ], such that other functions can more easily work with the data instead of having to re-parse it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After thinking about it, that is a good idea. But I think this should be stored in an internal variable like _address with the string representation stored in address. The reason being that the general use case for the IP address is in it's string representation. But yeah having the internal representation as a list of ints is probably the best state to store it internally.

cidr = "192.168.0.1/24";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cidr = "192.168.0.1/24";

Doesn't seem to be in the result.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You sure? I just created a test that fails to check the _parse function and it definitely is in there.

:!./tests.sh                                                                                                                                          
test case at ./tests.sh:138 failed: internal.ipv4._parse "192.168.0.1" "24" should have evaluated to "192.168.0.1":
"192.168.0.1"

but it evaluated to
{ address = "192.168.0.1"; cidr = "192.168.0.1/24"; prefixLength = "24"; subnetMask = "255.255.255.0"; }

shell returned 1

prefixLength = "24";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
prefixLength = "24";
prefixLength = 24;

Similarly

}
```

# Type

```
_parse :: String -> IPv4Address
```

# Arguments

- [address] An IPv4 address.
- [prefixLength] A prefix length.
*/
_parse = address: prefixLength: {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function doesn't actually do any parsing at all. Overall I think these internal functions could be refactored to this:

  • _parseIp :: String -> String -> Int
    Example:
    _parseIp "lib.network.ipv4.someFunction" "12.34.56.78"
    -> 203569230
    
    _parseIp "lib.network.ipv4.someFunction "12.34.56"
    -> error: lib.network.ipv4.someFunction: IPv4 address "12.34.56" is not of the correct form
  • _parsePrefix :: String -> String -> Int
    Example:
    _parsePrefix "lib.network.ipv4.someFunction" "16"
    -> 16
    
    _parsePrefix "lib.network.ipv4.someFunction" "160"
    -> error: lib.network.ipv4.someFunction: IPv4 prefix "160" is too large.
  • _parseCIDR :: String -> String -> { ip :: Int, prefix :: Int }
    Splits and calls _parseIp and _parsePrefix underneath

At this point, the public fromCidrString function can call _parseCIDR "lib.network.ipv4.fromCidrString" to get back two integers that are guaranteed to be valid. If they're not you'll get failure that also indicates that function call failed (you can add extra context to the error messages as desired).

All the internal functions for generating data now don't need to do any validation themselves.

Copy link
Member Author

@djacu djacu Apr 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm having a hard time understanding your suggestion. What is the first argument for these parse functions supposed to do?

For example, in _parseIp "lib.network.ipv4.someFunction" "12.34.56.78", I don't understand what the intent of passing "lib.network.ipv4.someFunction" and how it would be used in the function.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what the intent of passing "lib.network.ipv4.someFunction" and how it would be used in the function.

AFAICT, it seems to be for error messages to only mention public functions (fromCidrString) instead of private ones (_parseCIDR). Using these, we can make sure users only get messages about the API of the library, and not implementation details.

c.f.

the public fromCidrString function can call _parseCIDR "lib.network.ipv4.fromCidrString"

cidr = concatStringsSep "/" [
address
prefixLength
];
inherit address prefixLength;
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider setting _type = "ipv4Address"; for runtime type checking. This can improve error messages when addresses end up in wrong places; useful even if this library doesn't check _type.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also reminds me that a type in the module system would be nice (for later perhaps).
It could parse both stringly definitions and pass through _type-ed definitions.

};
}
124 changes: 124 additions & 0 deletions lib/network/tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
#!/usr/bin/env bash

# Tests lib.network
# Run:
# [nixpkgs]$ lib/network/tests.sh

set -euo pipefail


die() {
# The second to last entry contains the line number of the top-level caller
lineIndex=$(( ${#BASH_LINENO[@]} - 2 ))
echo >&2 -e "test case at ${BASH_SOURCE[0]}:${BASH_LINENO[$lineIndex]} failed:" "$@"
exit 1
}


if test -n "${TEST_LIB:-}"; then
NIX_PATH=nixpkgs="$(dirname "$TEST_LIB")"
else
NIX_PATH=nixpkgs="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.."; pwd)"
fi
export NIX_PATH


tmp="$(mktemp -d)"
clean_up() {
rm -rf "$tmp"
}
trap clean_up EXIT SIGINT SIGTERM
work="$tmp/work"
mkdir "$work"
cd "$work"

prefixExpression='
let
lib = import <nixpkgs/lib>;
internal = import <nixpkgs/lib/network/internal.nix> {
inherit lib;
};
in
with lib;
with lib.network;
'

# Check that two nix expression successfully evaluate to the same value.
# The expressions have `lib.fileset` in scope.
# Usage: expectEqual NIX NIX
expectEqual() {
local actualExpr=$1
local expectedExpr=$2
if actualResult=$(nix-instantiate --eval --strict --show-trace 2>"$tmp"/actualStderr \
--expr "$prefixExpression ($actualExpr)"); then
actualExitCode=$?
else
actualExitCode=$?
fi
actualStderr=$(< "$tmp"/actualStderr)

if expectedResult=$(nix-instantiate --eval --strict --show-trace 2>"$tmp"/expectedStderr \
--expr "$prefixExpression ($expectedExpr)"); then
expectedExitCode=$?
else
expectedExitCode=$?
fi
expectedStderr=$(< "$tmp"/expectedStderr)

if [[ "$actualExitCode" != "$expectedExitCode" ]]; then
echo "$actualStderr" >&2
echo "$actualResult" >&2
die "$actualExpr should have exited with $expectedExitCode, but it exited with $actualExitCode"
fi

if [[ "$actualResult" != "$expectedResult" ]]; then
die "$actualExpr should have evaluated to $expectedExpr:\n$expectedResult\n\nbut it evaluated to\n$actualResult"
fi

if [[ "$actualStderr" != "$expectedStderr" ]]; then
die "$actualExpr should have had this on stderr:\n$expectedStderr\n\nbut it was\n$actualStderr"
fi
}


# Check that a nix expression fails to evaluate (strictly, read-write-mode).
# And check the received stderr against a regex
# The expression has `lib.network` in scope.
# Usage: expectFailure NIX REGEX
expectFailure() {
local expr=$1
local expectedErrorRegex=$2
if result=$(nix-instantiate --eval --strict --read-write-mode --show-trace 2>"$tmp/stderr" \
--expr "$prefixExpression $expr"); then
die "$expr evaluated successfully to $result, but it was expected to fail"
fi
stderr=$(<"$tmp/stderr")
if [[ ! "$stderr" =~ $expectedErrorRegex ]]; then
die "$expr should have errored with this regex pattern:\n\n$expectedErrorRegex\n\nbut this was the actual error:\n\n$stderr"
fi
}


# Test basic cases for ingesting a CIDR string
expectEqual '(ipv4.fromCidrString "192.168.0.1/24").cidr' '"192.168.0.1/24"'
expectEqual '(ipv4.fromCidrString "192.168.0.1/24").address' '"192.168.0.1"'
expectEqual '(ipv4.fromCidrString "192.168.0.1/24").prefixLength' '"24"'

expectEqual '(ipv4.fromCidrString "192.168.0.1").prefixLength' '"32"'

# Test cases for bad IP addresses
expectFailure 'ipv4.fromCidrString "192.168.0./24"' 'lib.network.ipv4: CIDR 192.168.0./24 has an empty octet.'
expectFailure 'ipv4.fromCidrString "192.168.0/24"' 'lib.network.ipv4: CIDR 192.168.0/24 is not of the correct form.'
expectFailure 'ipv4.fromCidrString "192.168.0.256/24"' 'lib.network.ipv4: CIDR 192.168.0.256/24 has an out of bounds octet.'
expectFailure 'ipv4.fromCidrString "192.168.0.-1/24"' 'lib.network.ipv4: CIDR 192.168.0.-1/24 has an out of bounds octet.'

# Test cases for bad prefix length
expectFailure 'ipv4.fromCidrString "192.168.0.1/"' 'lib.network.ipv4: CIDR 192.168.0.1/ has no prefix length.'
expectFailure 'ipv4.fromCidrString "192.168.0.1/33"' 'lib.network.ipv4: CIDR 192.168.0.1/33 has an out of bounds prefix length, 33.'
expectFailure 'ipv4.fromCidrString "192.168.0.1/-1"' 'lib.network.ipv4: CIDR 192.168.0.1/-1 has an out of bounds prefix length, -1.'
expectFailure 'ipv4.fromCidrString "192.168.0.1/24/bad"' 'lib.network.ipv4: Could not verify prefix length for CIDR 192.168.0.1/24/bad'

# Test pow function
expectEqual 'internal.common.pow 2 0' '1'
expectEqual 'internal.common.pow 2 3' '8'
expectFailure 'internal.common.pow 2 (-1)' 'lib.network.pow: Exponent cannot be negative.'
3 changes: 3 additions & 0 deletions lib/tests/test-with-nix.nix
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ pkgs.runCommand "nixpkgs-lib-tests-nix-${nix.version}" {
echo "Running lib/fileset/tests.sh"
TEST_LIB=$PWD/lib bash lib/fileset/tests.sh

echo "Running lib/network/tests.sh"
TEST_LIB=$PWD/lib bash lib/network/tests.sh

echo "Running lib/tests/systems.nix"
[[ $(nix-instantiate --eval --strict lib/tests/systems.nix | tee /dev/stderr) == '[ ]' ]];

Expand Down