Using age as a git clean/smudge filter?

[magthe]

I'm having some troubles with using age as a filter in git. I sent a question to the Git User list but thought I might have more luck here. It can be summarised as

The config of the filter works as intended, but git diff believes that filtered file is modified in a brand new clone.

What follows now is the text I posted to the Git User list, but with some markdown to make it nicer.

---

I'm trying out age as a filter for encrypting files in a git repo but I must be missing something because every new clone thinks the encrypted file has changed, and if I commit that change then every other clone sees a diff after pulling in the change.

The main reason I want to try out age is that it can make use of SSH keys for encryption, which makes it a bit nicer than something like git-crypt.

I'm setting it up like this:

❯ cat .gitattributes
*.secret filter=age

❯ git config -l --local|grep filter.age
filter.age.smudge=age --decrypt -i ~/.ssh/id_ed25519 -
filter.age.clean=age --encrypt -R ~/.ssh/id_ed25519.pub -

Here's a sequence setting up a first repo:

❯ git init
Initialized empty Git repository in /home/user/tmp/age-0/.git/

❯ echo '*.secret filter=age' > .gitattributes

❯ git config --local --add filter.age.smudge "age --decrypt -i ~/.ssh/id_ed25519
-"

❯ git config --local --add filter.age.clean "age --encrypt -R
~/.ssh/id_ed25519.pub -"

❯ echo "a secret" > foo.secret

❯ echo "not a secret" > bar.txt

❯ git add .gitattributes bar.txt foo.secret

❯ git commit -m 'The first commit'
[main (root-commit) ae75577] The first commit
3 files changed, 2 insertions(+)
create mode 100644 .gitattributes
create mode 100644 bar.txt
create mode 100644 foo.secret

Now I can make a clone:

❯ cd ..

❯ git clone age-0 age-1
Cloning into 'age-1'...
done.

❯ cd age-1

❯ git ls-files
.gitattributes
bar.txt
foo.secret

❯ cat foo.secret
age-encryption.org/v1
-> ssh-ed25519 ozAWLA ReSnu8CTgPgnuKUMvG8PWTcc7Lr5IHkKaWc6k4Hfsms
dHsdERPHdsdOQluzyeeRamfjIrmsc2pQ+lhwLlt/0no
--- aHijNp3L2/0MeE/EXWwVhVwyv1uBYW1Ake055jico5M
WF}`YqBO7Ԏwߨ%

So far so good. The file is encrypted. Now I configure the filter the same way
and make sure the file is decrypted:

❯ git config --local --add filter.age.smudge "age --decrypt -i ~/.ssh/id_ed25519
-"

age-1 on  main
❯ git config --local --add filter.age.clean "age --encrypt -R
~/.ssh/id_ed25519.pub -"

❯ rm foo.secret

❯ git reset --hard HEAD
HEAD is now at ae75577 The first commit

❯ cat foo.secret
a secret

Now comes the problem, git thinks the file with secrets has been changed when it
really hasn't:

❯ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
 (use "git add <file>..." to update what will be committed)
 (use "git restore <file>..." to discard changes in working   directory)
       modified:   foo.secret

no changes added to commit (use "git add" and/or "git commit -a")

❯ git diff
diff --git a/foo.secret b/foo.secret
index 2de33ca..18e4331 100644
Binary files a/foo.secret and b/foo.secret differ

❯ md5sum foo.secret ../age-0/foo.secret
6046316bf834dbdf83a5be74be6fd2ac  foo.secret
6046316bf834dbdf83a5be74be6fd2ac  ../age-0/foo.secret

This isn't what I expected. What's wrong with my setup, what am I missing?

[magthe]

Oh, I should probably mention that I'm aware of agebox, but AFAICS it doesn't really work with git but rather in parallel with it.

[str4d]

Your problem is most likely with your filter.age.clean implementation.

age file encryption is not deterministic. age --encrypt called twice with the same plaintext mad recipients will produce different ciphertexts.

I'm guessing however that filter.*.smudge is intended for invertible transformations, with filter.*.clean being the inverse.

You either need to figure out how to massage the filter APIs to support non-deterministic / non-invertible transformations (e.g. re-decrypt the old version in filter.*.clean and check they don't match before encrypting), or you will need to use deterministic symmetric encryption (which comes with a host of caveats, some of which might be okay depending on how you're planning on using this).

[alerque]

I don't think you need cat there, I think you need a decryption operation.

[magthe]

No, as I mentioned, when I set diff.age.textconv=my-shell-script to experiment with it I found that it always got the clear text of the file.

[supermarin]

FWIW I have it implemented to handle both ASCII and bin, and am still facing this problem. @str4d 's comment explains it all - encryption is non deterministic. Encrypted blobs in index differ, but decrypted text is identical. Want to solve it the right way vs ditch age&filters for this, but I'm kind of puzzled ATM.

[supermarin]

On the other hand, gpg also encrypts non-deterministically, so this must be something we're both doing wrong 🤷

[prskr]

super smart guy here solved the issue by

1. decrypting the file with the given path at HEAD
2. hashing the decrypting file
3. hashing the same file in the working tree
4. if the has matches, return the already encrypted object again to avoid confusing Git

quite brilliant I must say (it wasn't me 😄 )

[prskr]

Hey, just stumbled upon this. I'm currently building something I'm calling (as working title) git-age that tries to solve this problem.

I only started last week so it's more of a PoC than a production ready CLI so far - I'm not satisfied yet with the test coverage 🙈 but it kinda works so give it a try and let me know if you're missing anything (besides more docs 😂 I'm unfortunately aware but I'm trying).

With a bit of luck I'll be able to provide you with a release and pre-compiled binaries some time next week.

@FiloSottile any feelings about the name? Shall I change it to avoid the impression it's tight to age itself?

[george-angel]

We also have an application that uses clean/smudge filters to encrypt files: https://github.com/uw-labs/strongbox

We have been using it for 7y+ but was initially written using SIV encryption. Recently added age support, and @prskr 's SOPS link have been very useful to address lack of determinism - thank you for finding that 🙇