Add variable expansion to URLs for values #169

jkroepke · 2021-11-22T18:35:29Z

What this PR does / why we need it:

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

github-actions · 2021-11-22T18:36:09Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck -x' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- scripts/lib/common.sh.orig
+++ scripts/lib/common.sh
@@ -81,16 +81,16 @@
 
 # https://stackoverflow.com/a/40167919
 expand_vars_strict() {
-  while IFS= read -r line || [ -n "${line}" ]; do  # the `||` clause ensures that the last line is read even if it doesn't end with \n
-    # Escape ALL chars. that could trigger an expansion..
-    lineEscaped=$(
-        printf %s "$line" | \
-        tr '`([$' '\1\2\3\4' | \
-        # ... then selectively reenable ${ references
-        sed -e 's/'"$(printf '\x4')"'{/\${/g' | \
-        # Finally, escape embedded double quotes to preserve them.
-        sed -e 's/"/\\\"/g' \
-    )
-    eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$'
-  done
+    while IFS= read -r line || [ -n "${line}" ]; do # the `||` clause ensures that the last line is read even if it doesn't end with \n
+        # Escape ALL chars. that could trigger an expansion..
+        lineEscaped=$(
+            printf %s "$line" |
+                tr '`([$' '\1\2\3\4' |
+                # ... then selectively reenable ${ references
+                sed -e 's/'"$(printf '\x4')"'{/\${/g' |
+                # Finally, escape embedded double quotes to preserve them.
+                sed -e 's/"/\\\"/g'
+        )
+        eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$'
+    done
 }
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions · 2021-11-22T18:46:03Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck -x' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- scripts/lib/expand_vars_strict.sh.orig
+++ scripts/lib/expand_vars_strict.sh
@@ -5,15 +5,15 @@
 # https://stackoverflow.com/a/40167919
 expand_vars_strict() {
     _x4=$(printf '\x4')
-    while IFS= read -r line || [ -n "${line}" ]; do  # the `||` clause ensures that the last line is read even if it doesn't end with \n
+    while IFS= read -r line || [ -n "${line}" ]; do # the `||` clause ensures that the last line is read even if it doesn't end with \n
         # Escape ALL chars. that could trigger an expansion..
         lineEscaped=$(
-            printf %s "$line" | \
-            tr '`([$' '\1\2\3\4' | \
-            # ... then selectively reenable ${ references
-            sed -e "s/$_x4{/\${/g" | \
-            # Finally, escape embedded double quotes to preserve them.
-            sed -e 's/"/\\\"/g' \
+            printf %s "$line" |
+                tr '`([$' '\1\2\3\4' |
+                # ... then selectively reenable ${ references
+                sed -e "s/$_x4{/\${/g" |
+                # Finally, escape embedded double quotes to preserve them.
+                sed -e 's/"/\\\"/g'
         )
         eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$'
     done
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

codecov · 2021-11-22T18:46:50Z

Codecov Report

Merging #169 (30cc2ea) into main (ad8787a) will decrease coverage by 0.27%.
The diff coverage is 75.00%.

@@            Coverage Diff             @@
##             main     #169      +/-   ##
==========================================
- Coverage   84.77%   84.50%   -0.28%     
==========================================
  Files          23       24       +1     
  Lines         611      626      +15     
==========================================
+ Hits          518      529      +11     
- Misses         93       97       +4

Impacted Files	Coverage Δ
scripts/lib/expand_vars_strict.sh	`60.00% <60.00%> (ø)`
scripts/lib/file/http.sh	`84.61% <100.00%> (+6.83%)`	⬆️
scripts/run.sh	`95.60% <100.00%> (+0.04%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad8787a...30cc2ea. Read the comment docs.

github-actions · 2021-11-22T18:47:53Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck -x' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- scripts/lib/expand_vars_strict.sh.orig
+++ scripts/lib/expand_vars_strict.sh
@@ -5,15 +5,15 @@
 # https://stackoverflow.com/a/40167919
 expand_vars_strict() {
     _x4=$(printf '\x4')
-    while IFS= read -r line || [ -n "${line}" ]; do  # the `||` clause ensures that the last line is read even if it doesn't end with \n
+    while IFS= read -r line || [ -n "${line}" ]; do # the `||` clause ensures that the last line is read even if it doesn't end with \n
         # Escape ALL chars. that could trigger an expansion..
         lineEscaped=$(
-            printf %s "$line" | \
-                tr '`([$' '\1\2\3\4' | \
+            printf %s "$line" |
+                tr '`([$' '\1\2\3\4' |
                 # ... then selectively reenable ${ references
-                sed -e "s/$_x4{/\${/g" | \
+                sed -e "s/$_x4{/\${/g" |
                 # Finally, escape embedded double quotes to preserve them.
-                sed -e 's/"/\\\"/g' \
+                sed -e 's/"/\\\"/g'
         )
         eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$'
     done
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions · 2021-11-22T18:51:01Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck -x' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- scripts/lib/expand_vars_strict.sh.orig
+++ scripts/lib/expand_vars_strict.sh
@@ -5,15 +5,15 @@
 # https://stackoverflow.com/a/40167919
 expand_vars_strict() {
     _x4=$(printf '\x4')
-    while IFS= read -r line || [ -n "${line}" ]; do  # the `||` clause ensures that the last line is read even if it doesn't end with \n
+    while IFS= read -r line || [ -n "${line}" ]; do # the `||` clause ensures that the last line is read even if it doesn't end with \n
         # Escape ALL chars. that could trigger an expansion..
         lineEscaped=$(
-            printf %s "$line" | \
-                tr '`([$' '\1\2\3\4' | \
+            printf %s "$line" |
+                tr '`([$' '\1\2\3\4' |
                 # ... then selectively reenable ${ references
-                sed -e "s/$_x4{/\${/g" | \
+                sed -e "s/$_x4{/\${/g" |
                 # Finally, escape embedded double quotes to preserve them.
-                sed -e 's/"/\\\"/g' \
+                sed -e 's/"/\\\"/g'
         )
         eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$'
     done
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

jkroepke · 2021-11-22T19:01:51Z

Hey @jmclean-starburst, is it possible that you can checkout this and test it on your setup?

jmclean-starburst · 2021-11-22T19:35:27Z

getting the below

error loading config: no matching creation rules found
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5401918]

jkroepke · 2021-11-22T20:10:59Z

Hm, it looks good in general here, because

no errors from curl, it means fetch the file was successful.
panic: runtime error: invalid memory address or nil pointer dereference
This is expectable. helm throws a gopanic, if loading the file was not successfully.
go panic with plugin downloader helm/helm#8935
error loading config: no matching creation rules found. This comes from sops maybe after fetching the content. I have no idea, why this comes. This error is expectable at encryption process, but not on the decryption process.

Set HELM_SECRETS_DEBUG=true help here to get deeper into this. This enable the shell builtin debugging.

jmclean-starburst · 2021-11-22T20:46:18Z

Not sure...the error is kicked right after sops:

+ _sops --decrypt --input-type yaml --output-type yaml /var/folders/pr/wrsgr_fd57bcnht422blbwq40000gn/T/tmp.9i0DFDTh/7unlRm
+ set -- --decrypt --input-type yaml --output-type yaml /var/folders/pr/wrsgr_fd57bcnht422blbwq40000gn/T/tmp.9i0DFDTh/7unlRm
+ sops --decrypt --input-type yaml --output-type yaml /var/folders/pr/wrsgr_fd57bcnht422blbwq40000gn/T/tmp.9i0DFDTh/7unlRm
error loading config: no matching creation rules found

FWIW - i pass in the url string being surrounded by single quotes (ie '), so something like:

HELM_SECRETS_URL_VARIABLE_EXPANSION=true HELM_SECRETS_DEBUG=true helm template mychart ./mychart -f 'secrets://https://${GITHUB_TOKEN}@raw.githubusercontent.com/org/repo/ref/pathtofile.yml'

I do this because I am trying to mimic what ArgoCD passes in.

jkroepke · 2021-11-22T21:09:27Z

Looks like getsops/sops#884

jmclean-starburst · 2021-11-22T21:11:13Z

seems like the only time you leverage the expansion is via the below, which doesnt actually update the $file variable used


    if ! _file_exists "$file"; then
        error 'File does not exist: %s\n' "${file}"
    fi

    real_file=$(_file_get "${file}")

I would have expected the real_file to fetch from the newly created url

jkroepke · 2021-11-22T21:17:16Z

_file_get is calling _file_http_get and the functions eval the URL, download the file and returns the path of the temporary file which holds the content of the curl command.

Then, the temporary file path real_file is passed to sops.

What happens, if you are using a invalid GITHUB_TOKEN? same error?

jmclean-starburst · 2021-11-22T21:20:10Z

a bad token gives the below after a curl command

curl: (22) The requested URL returned error: 404

Then I get the:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5401918]

jkroepke · 2021-11-22T21:22:58Z

This validates that the evaluation works in general.

Maybe we are hitting getsops/sops#884 here. I do a commit here which introduce a workaround for that case.

Otherwise, the encrypted yaml file from org/repo/ref/pathtofile.yml is strange.

jmclean-starburst · 2021-11-22T21:42:33Z

@jkroepke definitely seems like the issue is the sops tag above....i had a local .sops.yaml file with more specific rules. When I deleted it, it likely defaulted to your file which is more of a catch al, and it worked! thanks for all the work on this. we plan on using it HEAVILY ;)

jkroepke · 2021-11-22T21:56:24Z

Thanks for the feedback. Great to know that it works now.

and it worked!

I would like to know, if I the empty sops.yaml file is still needed, because it worked years fine without it.

Before merging this, I still needs to write some documentation about this feature.

jkroepke mentioned this pull request Nov 22, 2021

translate env vars if found within file string #168

Closed

1 task

jkroepke changed the title ~~Add variable expansion to URLs~~ Add variable expansion to URLs for values Nov 22, 2021

jkroepke force-pushed the var_expansion branch from 643f487 to 5b5e5bb Compare November 23, 2021 11:57

Add variable expansion

4a61e70

jkroepke force-pushed the var_expansion branch from 5b5e5bb to 4a61e70 Compare November 23, 2021 12:30

Added some documentation

30cc2ea

jkroepke merged commit 0dcbc7e into main Nov 23, 2021

jkroepke deleted the var_expansion branch November 23, 2021 13:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add variable expansion to URLs for values #169

Add variable expansion to URLs for values #169

jkroepke commented Nov 22, 2021 •

edited

Loading

github-actions bot commented Nov 22, 2021

github-actions bot commented Nov 22, 2021

codecov bot commented Nov 22, 2021 •

edited

Loading

github-actions bot commented Nov 22, 2021

github-actions bot commented Nov 22, 2021

jkroepke commented Nov 22, 2021

jmclean-starburst commented Nov 22, 2021

jkroepke commented Nov 22, 2021 •

edited

Loading

jmclean-starburst commented Nov 22, 2021 •

edited

Loading

jkroepke commented Nov 22, 2021

jmclean-starburst commented Nov 22, 2021

jkroepke commented Nov 22, 2021 •

edited

Loading

jmclean-starburst commented Nov 22, 2021

jkroepke commented Nov 22, 2021

jmclean-starburst commented Nov 22, 2021

jkroepke commented Nov 22, 2021

Add variable expansion to URLs for values #169

Add variable expansion to URLs for values #169

Conversation

jkroepke commented Nov 22, 2021 • edited Loading

github-actions bot commented Nov 22, 2021

sh-checker report

github-actions bot commented Nov 22, 2021

sh-checker report

codecov bot commented Nov 22, 2021 • edited Loading

Codecov Report

github-actions bot commented Nov 22, 2021

sh-checker report

github-actions bot commented Nov 22, 2021

sh-checker report

jkroepke commented Nov 22, 2021

jmclean-starburst commented Nov 22, 2021

jkroepke commented Nov 22, 2021 • edited Loading

jmclean-starburst commented Nov 22, 2021 • edited Loading

jkroepke commented Nov 22, 2021

jmclean-starburst commented Nov 22, 2021

jkroepke commented Nov 22, 2021 • edited Loading

jmclean-starburst commented Nov 22, 2021

jkroepke commented Nov 22, 2021

jmclean-starburst commented Nov 22, 2021

jkroepke commented Nov 22, 2021

jkroepke commented Nov 22, 2021 •

edited

Loading

`sh-checker report`

`sh-checker report`

codecov bot commented Nov 22, 2021 •

edited

Loading

`sh-checker report`

`sh-checker report`

jkroepke commented Nov 22, 2021 •

edited

Loading

jmclean-starburst commented Nov 22, 2021 •

edited

Loading

jkroepke commented Nov 22, 2021 •

edited

Loading