Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fileset.fileFilter: Restrict second argument to paths #267384

Merged
merged 1 commit into from
Nov 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions lib/fileset/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,21 @@ Arguments:
And it would be unclear how the library should behave if the one file wouldn't be added to the store:
`toSource { root = ./file.nix; fileset = <empty>; }` has no reasonable result because returing an empty store path wouldn't match the file type, and there's no way to have an empty file store path, whatever that would mean.

### `fileFilter` takes a path

The `fileFilter` function takes a path, and not a file set, as its second argument.

- (-) Makes it harder to compose functions, since the file set type, the return value, can't be passed to the function itself like `fileFilter predicate fileset`
- (+) It's still possible to use `intersection` to filter on file sets: `intersection fileset (fileFilter predicate ./.)`
- (-) This does need an extra `./.` argument that's not obvious
- (+) This could always be `/.` or the project directory, `intersection` will make it lazy
- (+) In the future this will allow `fileFilter` to support a predicate property like `subpath` and/or `components` in a reproducible way.
This wouldn't be possible if it took a file set, because file sets don't have a predictable absolute path.
- (-) What about the base path?
- (+) That can change depending on which files are included, so if it's used for `fileFilter`
it would change the `subpath`/`components` value depending on which files are included.
- (+) If necessary, this restriction can be relaxed later, the opposite wouldn't be possible

## To update in the future

Here's a list of places in the library that need to be updated in the future:
Expand Down
20 changes: 15 additions & 5 deletions lib/fileset/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -366,7 +366,7 @@ in {
type :: String,
...
} -> Bool)
-> FileSet
-> Path
-> FileSet

Example:
Expand Down Expand Up @@ -397,14 +397,24 @@ in {
Other attributes may be added in the future.
*/
predicate:
# The file set to filter based on the predicate function
fileset:
# The path whose files to filter
path:
if ! isFunction predicate then
throw ''
lib.fileset.fileFilter: First argument is of type ${typeOf predicate}, but it should be a function instead.''
else if ! isPath path then
if path._type or "" == "fileset" then
throw ''
lib.fileset.fileFilter: Second argument is a file set, but it should be a path instead.
If you need to filter files in a file set, use `intersection fileset (fileFilter pred ./.)` instead.''
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you need to filter files in a file set, use `intersection fileset (fileFilter pred ./.)` instead.''
If you need to filter files in a file set, use `intersection fileset (fileFilter pred ./.)` instead.''

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to keep this for now because it's consistent with other error messages in the library.

else
throw ''
lib.fileset.fileFilter: Second argument is of type ${typeOf path}, but it should be a path instead.''
else if ! pathExists path then
throw ''
lib.fileset.fileFilter: Second argument (${toString path}) is a path that does not exist.''
else
_fileFilter predicate
(_coerce "lib.fileset.fileFilter: Second argument" fileset);
_fileFilter predicate path;

/*
The file set containing all files that are in both of two given file sets.
Expand Down
33 changes: 18 additions & 15 deletions lib/fileset/internal.nix
Original file line number Diff line number Diff line change
Expand Up @@ -786,9 +786,9 @@ rec {
_differenceTree (path + "/${name}") lhsValue (rhs.${name} or null)
) (_directoryEntries path lhs);

# Filters all files in a file set based on a predicate
# Type: ({ name, type, ... } -> Bool) -> FileSet -> FileSet
_fileFilter = predicate: fileset:
# Filters all files in a path based on a predicate
# Type: ({ name, type, ... } -> Bool) -> Path -> FileSet
_fileFilter = predicate: root:
let
# Check the predicate for a single file
# Type: String -> String -> filesetTree
Expand All @@ -807,19 +807,22 @@ rec {

# Check the predicate for all files in a directory
# Type: Path -> filesetTree
fromDir = path: tree:
mapAttrs (name: subtree:
if isAttrs subtree || subtree == "directory" then
fromDir (path + "/${name}") subtree
else if subtree == null then
null
fromDir = path:
mapAttrs (name: type:
if type == "directory" then
fromDir (path + "/${name}")
else
fromFile name subtree
) (_directoryEntries path tree);
fromFile name type
) (readDir path);

rootType = pathType root;
in
if fileset._internalIsEmptyWithoutBase then
_emptyWithoutBase
if rootType == "directory" then
_create root (fromDir root)
else
_create fileset._internalBase
(fromDir fileset._internalBase fileset._internalTree);
# Single files are turned into a directory containing that file or nothing.
_create (dirOf root) {
${baseNameOf root} =
fromFile (baseNameOf root) rootType;
};
}
15 changes: 4 additions & 11 deletions lib/fileset/tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -813,14 +813,15 @@ checkFileset 'difference ./. ./b'
# The first argument needs to be a function
expectFailure 'fileFilter null (abort "this is not needed")' 'lib.fileset.fileFilter: First argument is of type null, but it should be a function instead.'

# The second argument can be a file set or an existing path
expectFailure 'fileFilter (file: abort "this is not needed") null' 'lib.fileset.fileFilter: Second argument is of type null, but it should be a file set or a path instead.'
# The second argument needs to be an existing path
expectFailure 'fileFilter (file: abort "this is not needed") _emptyWithoutBase' 'lib.fileset.fileFilter: Second argument is a file set, but it should be a path instead.
\s*If you need to filter files in a file set, use `intersection fileset \(fileFilter pred \./\.\)` instead.'
expectFailure 'fileFilter (file: abort "this is not needed") null' 'lib.fileset.fileFilter: Second argument is of type null, but it should be a path instead.'
expectFailure 'fileFilter (file: abort "this is not needed") ./a' 'lib.fileset.fileFilter: Second argument \('"$work"'/a\) is a path that does not exist.'

# The predicate is not called when there's no files
tree=()
checkFileset 'fileFilter (file: abort "this is not needed") ./.'
checkFileset 'fileFilter (file: abort "this is not needed") _emptyWithoutBase'

# The predicate must be able to handle extra attributes
touch a
Expand Down Expand Up @@ -882,14 +883,6 @@ checkFileset 'union ./c/a (fileFilter (file: assert file.name != "a"; true) ./.)
# but here we need to use ./c
checkFileset 'union (fileFilter (file: assert file.name != "a"; true) ./.) ./c'

# Also lazy, the filter isn't called on a filtered out path
tree=(
[a]=1
[b]=0
[c]=0
)
checkFileset 'fileFilter (file: assert file.name != "c"; file.name == "a") (difference ./. ./c)'

# Make sure single files are filtered correctly
tree=(
[a]=1
Expand Down