creation_rules from can't be matched when encrypting from stdin

[adrian-gierakowski]

given a project with following structure:

.
├── .sops.yaml
└── secrets
    └── test.json

with .sops.yaml content:

creation_rules:
  - path_regex: secrets/.*
    gcp_kms: projects/YOUR_POJECT/locations/global/keyRings/YOUR_RING/cryptoKeys/YOUR_KEY

I am not able to encrypt data from stdin into the secrets directory, for example

cat secrets/test.json | sops --input-type json --output-type json --output secrets/test.encrypted.json -e /dev/stdin

gives me:

error loading config: no matching creation rules found

I am guessing that sops is trying to match /dev/stdin against the creation_rules from the config file (changing path_regex: secrets/.* to path_regex: .* makes it work, but that is obviously not a solution).

Here's a sample repo which can be used to reproduce the error:
https://github.com/adrian-gierakowski/sops-encrypt-from-stdin

Possible solutions

1. match against value of --output (converted to absolute path, or path relative to config file) if present
2. add a new flag which would take a path that could be used to match against creation rules

The latter would be more universal as it would allow use cases where --output is not specified

As a side note: I stumbled upon this issue while trying to write a script to re-encrypt all files in a directory after changing the creation_rules, and wanted to avoid temporarily storing unencrypted data to disk while doing it.

[autrilla]

I am guessing that sops is trying to match /dev/stdin against the creation_rules from the config file

You guessed right.

As a side note: I stumbled upon this issue while trying to write a script to re-encrypt all files in a directory after changing the creation_rules, and wanted to avoid temporarily storing unencrypted data to disk while doing it.

Sounds like you might want sops updatekeys.

[autrilla]

This looks more like a feature request than a bug. I'm not sure this is a feature we actually want, since your use case is something SOPS already provides out of the box.

[adrian-gierakowski]

I am guessing that sops is trying to match /dev/stdin against the creation_rules from the config file

You guessed right.

As a side note: I stumbled upon this issue while trying to write a script to re-encrypt all files in a directory after changing the creation_rules, and wanted to avoid temporarily storing unencrypted data to disk while doing it.

Sounds like you might want sops updatekeys.

Hi @autrilla, thanks for pointing this out. I actually initially tried updatekeys but it failed with the same error. The problem was that I'd first cd into the secrets dir and then call sops updatekeys test.json so the file path would fail to match secrets/.*. This can be worked around by doing sops updatekeys "$(pwd)/test.json" though.

This looks more like a feature request than a bug. I'm not sure this is a feature we actually want, since your use case is something SOPS already provides out of the box.

yes, I guess this is a feature request, but I'd say that even though updatekeys caters to the particular use case I brought up above, there are others which it does not. For example, I'm generating some credentials with terraform and would like to do something like:

terraform output private_key | sops --output secrets/private_key -e /dev/stdin

but with given the project structure as described above I'm not able to do it and need to resort to a two step process which risks leaving unencrypted secrets lying around:

terraform output private_key > secrets/private_key
sops -e -i secrets/private_key

[marcbachmann]

It would be nice if we could infer the path based on the --input-type.
My use case is that I want to use stdin or maybe add a new --file-name parameter.

echo foo: bar | sops --input-type somefile.enc.yaml -e /dev/stdin

That could then respect the path_regex and the creation rules

[stealthybox]

Hi friends, I think this might be a bug.

I ran into this problem trying to run a build of a ytt project that produces a Kubernetes manifest containing Secrets.
The YAML file being created is new, and stdin is sensitive generated build output:

# example
ytt -f ./config -f ./overlays --data-values-file <(sops --decrypt values.enc.yaml) \
  | sops --output ../k8s/ytt-build.enc.yaml --encrypt /dev/stdin

This fails to match my creation_rules for yaml files.

@autrilla the current path_regex behavior really surprised me in this case.

sops is trying to match /dev/stdin against the creation_rules from the config file

Shouldn't creation rules be matching on the path of the output file if possible?

The input path is a good fallback choice, if the output is defaulting to stdout.
In the --in-place case, they happen to be the same, but we implicitly know the output path.
However when --output is explicitly set, that seems to clearly indicate the name of the file sops will write to.

That's one reason I used the --output flag instead of redirecting output via the shell.
I assumed sops would see this flag set and trigger my creation rules.

Does anyone else agree this feels buggy?

Would the project accept a patch to amend the path matching behavior to consider --output as the final override?
Thanks, cheers.

[stealthybox]

--ouput was introduced in #387

[simonfxr]

I use a workaround that works for me:

some-command | EDITOR="cp -f /dev/stdin" sops secrets.yaml

[adrian-gierakowski]

Thanks @simonfxr, nice hack!

I’ve decided to abandon the use of .sops.yaml and instead use a wrapper around sops which decides which keys to use and passes it to sops as command line argument

[felixfontein]

@autrilla the current path_regex behavior really surprised me in this case.

sops is trying to match /dev/stdin against the creation_rules from the config file

Shouldn't creation rules be matching on the path of the output file if possible?

The input path is a good fallback choice, if the output is defaulting to stdout. In the --in-place case, they happen to be the same, but we implicitly know the output path. However when --output is explicitly set, that seems to clearly indicate the name of the file sops will write to.

That's one reason I used the --output flag instead of redirecting output via the shell. I assumed sops would see this flag set and trigger my creation rules.

Does anyone else agree this feels buggy?

Would the project accept a patch to amend the path matching behavior to consider --output as the final override? Thanks, cheers.

I agree that --output should be used instead of the input filename, if it is provided. Unfortunately this is a somewhat breaking change. @ajvb what do you think?

[felixfontein]

There's the case where the input is /dev/stdin and --output is not specified (which happens in #52). I've been thinking a bit about how to solve that in an elegant way (specifying --output is not an option, since output to stdout is needed), without having to find and parse and interpret (!) .sops.yaml manually.

How about having a new parameter --output-path (the name really isn't good, would be happy about any better suggestion!) that uses the provided path for determining the rules, without actually using that path for writing the result? This would avoid a breaking change, make the OP's use case a bit more complex (have to specify --output-path secrets/test.encrypted.json next to --output secrets/test.encrypted.json), but would also solve my case (stdin → stdout using rules for a specific path in the filesystem).

What do you think? Especially the new maintainers, @hiddeco @sabre1041 @onedr0p @devstein. If anyone has a better idea that also covers my use-case, I'm totally open for that as well :) (I'll also try to implement it.)

[felixfontein]

I created #1332 to implement a --filename-override option.

[adrian-gierakowski]

🎉