Profile scripts add duplicate entries to PATH in subshells
#5950
Comments
|
A little relevant discussion (and my current PATH-deduplication workarounds): #5298 |
|
I solved it with two lines in my bashrc which work universally: DEDUPE_PATH="$(printf %s "$PATH" | awk -v RS=: '{ if (!arr[$0]++) {printf("%s%s",!ln++?"":":",$0)}}')"
export PATH=$DEDUPE_PATH |
|
Pretty clever! The ternary in the printf confused me for a bit. I put a similar deduplicator in the same thread, though I didn't care for the extra ms added to my shell spawn time. In my testing you can shave off a ms or so by using a herestring instead of piping from printf (and I've added a little more spacing, which helped me understand what was going on): Also, the version I'm using is no slower and included for comparison: $ cat run.sh
#!/bin/bash
orig() {
printf %s "${PATH}" | awk -v RS=: '{ if (!arr[$0]++) {printf("%s%s",!ln++?"":":",$0)}}'
}
herestring() {
awk -v RS=: '{ if (!arr[$0]++) {printf("%s%s",!ln++?"":":",$0)}}' <<< "${PATH}"
}
mine() {
env --ignore-environment HOME="${HOME}" sh -c '
source /etc/static/bashrc
echo "${PATH}"
'
}
if [[ -n "$(herestring)" ]] && [[ "$(orig)" = "$(herestring)" ]]; then
export -f orig herestring mine
hyperfine -m 500 orig
hyperfine -m 500 herestring
hyperfine -m 500 mine
fi
$ ./run.sh
Benchmark 1: orig
Time (mean ± σ): 4.5 ms ± 0.5 ms [User: 1.9 ms, System: 1.3 ms]
Range (min … max): 3.8 ms … 8.4 ms 500 runs
...
Benchmark 1: herestring
Time (mean ± σ): 3.8 ms ± 0.4 ms [User: 1.9 ms, System: 1.3 ms]
Range (min … max): 3.3 ms … 7.2 ms 500 runs
...
Benchmark 1: mine
Time (mean ± σ): 3.7 ms ± 0.4 ms [User: 0.9 ms, System: 1.3 ms]
Range (min … max): 2.4 ms … 8.6 ms 500 runs |
|
You could try checking @n8henrie That said, if you're using nix-darwin, you shouldn't be sourcing the nix profile script at all. Your So if you're sourcing the nix profile separately you should clean that up. If you had a single-user install on your machine then it probably added the single-user nix profile to your This is of course assuming that |
Thanks @lilyball -- I don't think I am: #5298 (comment) I'm sourcing I use tmux, and my EDIT: For example, This led to obvious problems with nix and nix-darwin trying to decide whether or not the PATH was correct by looking at In the end, sourcing ¯\_(ツ)_/¯ |
|
I'm also curious about the speed of doing a bash-native deduplication, something like newpath=:
while read -d : path; do
if [[ $newpath != *:$path:* ]]; then
newpath+=$path:
fi
done <<<"$PATH:"
newpath=${newpath#:}
PATH=${newpath%:}This won't stop the profile script from potentially moving the nix paths in front of anything else of course, but it should behave like the That said, we really should just fix the profile scripts to be idempotent. |
|
Good call! $ cat run.sh
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ File: run.sh
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ #!/bin/bash
2 │
3 │ orig() {
4 │ printf %s "${PATH}" | awk -v RS=: '{ if (!arr[$0]++) {printf("%s%s",!ln++?"":":",$0)}}'
5 │ }
6 │
7 │ herestring() {
8 │ awk -v RS=: '{ if (!arr[$0]++) {printf("%s%s",!ln++?"":":",$0)}}' <<< "${PATH}"
9 │ }
10 │
11 │ mine() {
12 │ env --ignore-environment HOME="${HOME}" sh -c '
13 │ source /etc/static/bashrc
14 │ echo "${PATH}"
15 │ '
16 │ }
17 │
18 │ bashnative() {
19 │ newpath=:
20 │ while read -r -d : path; do
21 │ if [[ "${newpath}" != *:"${path}":* ]]; then
22 │ newpath+=${path}:
23 │ fi
24 │ done <<< "${PATH}:"
25 │ newpath=${newpath#:}
26 │ echo "${newpath%:}"
27 │ }
28 │
29 │ if [[ -n "$(herestring)" ]] && [[ "$(orig)" = "$(herestring)" ]] && [[ "$(bashnative)" = "$(orig)" ]]; then
30 │ export -f orig herestring mine bashnative
31 │ hyperfine -m 500 orig
32 │ hyperfine -m 500 herestring
33 │ hyperfine -m 500 mine
34 │ hyperfine -m 500 bashnative
35 │ fi
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
$ ./run.sh
Benchmark 1: orig
Time (mean ± σ): 4.7 ms ± 0.3 ms [User: 2.0 ms, System: 1.5 ms]
Range (min … max): 3.9 ms … 7.5 ms 500 runs
...
Benchmark 1: herestring
Time (mean ± σ): 4.1 ms ± 0.2 ms [User: 2.0 ms, System: 1.3 ms]
Range (min … max): 3.8 ms … 6.5 ms 500 runs
...
Benchmark 1: mine
Time (mean ± σ): 3.7 ms ± 0.4 ms [User: 1.0 ms, System: 1.4 ms]
Range (min … max): 2.8 ms … 6.2 ms 500 runs
...
Benchmark 1: bashnative
Time (mean ± σ): 1.7 ms ± 0.1 ms [User: 1.9 ms, System: 0.5 ms]
Range (min … max): 1.5 ms … 2.8 ms 522 runs
... |
|
It seems this issue could be solved similar to how Cargo does it. This line could become something like case ":$PATH:" in
*:"$HOME/.nix-profile/bin:@localstatedir@/nix/profiles/default/bin":*)
;;
*)
export PATH="$HOME/.nix-profile/bin:@localstatedir@/nix/profiles/default/bin:$PATH"
;;
esac |
For my usecase I also want to run it in the login shell because I already have duped PATHs at that level.
Ordering for PATH is super important for reboot traps, safe-rm and sudo wrappers. |
|
My strategy of "starting
On either of these, using So I think the "forgiveness over permission" approach of cleaning duplicates at the end is probably a better idea, will try out @lilyball's recommendation. |
|
By no longer creating the $ which -a bash
/run/current-system/sw/bin/bash
/bin/bash
$ bash -c 'which -a bash'
/run/current-system/sw/bin/bash
/bin/bash
$ bash --login -c 'which -a bash'
/bin/bash
/run/current-system/sw/bin/bashLogin shells now "see" that the nix is already in the environment, and I think the MacOS EDIT: Fix for now by having tmux not use a login shell ( |
|
Another approach to the problem on macOS with zsh. I try to avoid manipulating
This works well even with tmux. my $PATH are the same in terminal on startup and in tmux. no duplicates
for example, given an existing when a shell is spawned the resulting $PATH is Perhaps the way to fix the dup path issue is to utilise |
|
I used the Arch Linux package, so your mileage may vary, but this works for me. You may want to check your In your In your Bash or Zsh profile: Why the profile.d script doesn't check the existence of |
|
This is a pretty big issue in my opinion. If I have a python environment loaded up that I want to be persisted through an entire tmux session, I can't do that because |
|
A bash-native workaround is to add this to a home-manager config: programs.bash.bashrcExtra = ''
NIX_PATHS="$HOME/.nix-profile/bin:/nix/var/nix/profiles/default/bin:"
NEWPATH=''${PATH/$NIX_PATHS}
while [[ $NEWPATH =~ $NIX_PATHS ]]; do
PATH=$NEWPATH
NEWPATH=''${NEWPATH/$NIX_PATHS}
done
''Simply removes the paths string until it's gone, leaving the result with only one match as the |
|
similar problem happens in zsh, added this as a workarround in zshrc NIX_PATHS=$(echo $PATH | tr ':' '\n' | grep "/nix/" | tail -n +2 | tr '\n' ':')
if [[ $NIX_PATHS ]]; then
PATH=$NIX_PATHS$PATH
fi |
|
I have duplicated nix bin paths as well. There are several criterias my solution has to respect while solving the problem:
Here is my solution when placed in a separated script ( #!/bin/sh
keep_chosen_lines_unique () {
# stdin: lines
local chosen_lines=$1
local awk_code='
BEGIN {
'"$( \
echo "$chosen_lines" \
| tr ' ' '\n' \
| sed 's|\(.*\)|is_presents["\1"]="false"|')"'
} {
is_present=is_presents[$0]
if (length(is_present) > 0) {
if (is_present == "false") {
print $0
is_presents[$0]="true"
}
} else {
print $0
}
}
'
# stdin
awk "$awk_code"
}
match () {
local pattern=$1
local string=$2
eval "
case \"\$string\" in
$pattern) return 0 ;;
esac"
return 1
}
insert_before () {
# stdin: lines
local section=$1
local pattern=$2
while read line; do
if match "$pattern" "$line"; then
echo "$section"
echo "$line"
break
else
echo "$line"
fi
done # stdin
cat # stdin
}
get_nix_bin_paths () {
local nix_profiles=$1
if [ -z "${nix_profiles}" ]; then
return 0
fi
local output=$( \
echo "x $nix_profiles" \
| tr ' ' '\n' \
| tac \
| sed 's|$|/bin|' \
| tr '\n' ':')
echo "${output%:x/bin:}"
}
insert_unique_paths_after_home () {
local paths=$1
local new_paths=$2
if [ -n "${paths}" ]; then
if [ -n "${new_paths}" ]; then
local new_lines=$( \
echo "$new_paths" \
| tr ':' '\n' \
| tac)
local output=$( \
echo "${new_paths}:${paths}:x" \
| tr ':' '\n' \
| tac \
| insert_before "$new_lines" "$HOME/*" \
| keep_chosen_lines_unique "$new_lines" \
| tac \
| tr '\n' ':')
echo "${output%:x:}"
else
echo "$paths"
fi
else
if [ -n "${new_paths}" ]; then
echo "$new_paths"
fi
fi
}
test_insert_unique_paths_after_home () {
local nix_profiles=$1
local paths=$2
local nix_bin_paths=$(get_nix_bin_paths "$nix_profiles")
local output=$(insert_unique_paths_after_home "$paths" "$nix_bin_paths")
echo "input:"
echo "$paths"
echo
echo "output:"
echo "$output"
echo
}
test_all () {
local nix_profiles="/nix/var/nix/profiles/default $HOME/.nix-profile"
echo "Empty PATH"
test_insert_unique_paths_after_home "$nix_profiles" ""
echo
echo "PATH without $HOME/* + trailing empty paths"
test_insert_unique_paths_after_home "$nix_profiles" \
"/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:::"
echo
echo "PATH with repeated $HOME/bin + trailing empty paths"
test_insert_unique_paths_after_home "$nix_profiles" \
"$HOME/bin:/usr/local/bin:$HOME/bin:/usr/bin:/bin:/usr/local/games:/usr/games:::"
echo
echo "PATH with repeated $HOME/bin + trailing empty paths + repeated nix bin paths"
local nix_bin_paths=$(get_nix_bin_paths "$nix_profiles")
test_insert_unique_paths_after_home "$nix_profiles" \
"$nix_bin_paths:$HOME/bin:/usr/local/bin:$HOME/bin:$nix_bin_paths:/usr/bin:/bin:/usr/local/games:/usr/games:::"
echo
}
"$@"
To test do: Here is my result: |
Describe the bug
The Nix profile scripts unconditionally add directories to
PATHwithout checking if those directories are already there. The multi-user profile script has a guard variable to prevent it from running multiple times, but this guard variable is not exported and so does not affect subshell behavior. The single-user profile script doesn't even have a guard variable.The effect of this is that running an interactive subshell (if multi-user, or a login subshell if single-user) will source the profile script and add the directories to
PATHagain even though they're already there.Steps To Reproduce
$PATHincludes your nix profile bin directories.$PATHcontainsExpected behavior
$PATHshould contain your nix profile bin directories exactly once in both the parent shell and the subshell.Additional context
I have not actually reproduced this problem locally, as I am not currently set up to test standalone Nix installs (the machines I have access to use either NixOS or nix-darwin and I'm not prepared to run a VM at the moment). But this can be verified by reading the profile scripts, and this issue was originally reported to me at lilyball/nix-env.fish#11 in my fish plugin that imports the bash profile.
One potential fix is to just export the
__ETC_PROFILE_NIX_SOURCEDguard variable (and add this to the single-user profile too as that doesn't even have a guard variable right now), but I don't think that's a good fix assudowill preservePATHbut not the other variables (and especially not the guard variable) and so running an interactive or login shell viasudowould still add the duplicatePATHentries.Given that, I think the best approach is to just make sure the profile script is idempotent. I should be able to source it multiple times in a row (unsetting the guard variable each time) and end up with the same environment that I do if I only source it once. The guard variable still has a purpose with this approach, both as an optimization and to prevent re-sourcing the profile from blowing away any subsequent environment modifications (the installer sources the profile from potentially multiple files involved in shell initialization, and I might update the earliest such file to change the environment and so the subsequent re-sourcing of the profile shouldn't blow that away). Making the script idempotent simply requires making it check if the entries are in
PATHbefore setting them, as all other environment changes it does are already idempotent, though it's probably also worth putting a comment in the file noting that it should be idempotent as a caution for anyone modifying it in the future.The text was updated successfully, but these errors were encountered: